PHP tip: How to strip HTML tags, scripts, and styles from a web page

The HTML tags on a web page must be stripped away to get clean text for a PHP search engine, keyword extractor, or some other page analysis tool. PHP's standard strip_tags( ) function will do part of the job, but you need to strip out styles, scripts, embedded objects, and other unwanted page code first. This tip shows how.

This article is both an independent article and part of an article series on How to extract keywords from a web page. The rest of the series looks at how to get a web page from a web server, get the page's content type, convert to UTF-8, strip away punctuation, symbol characters, and numbers, and break a page down into a keyword list.

Code

PHP's handy strip_tags( ) function removes HTML tags that look like <word...>, <word.../>, or </word>. However, it doesn't understand the tags it's removing. It will blindly remove the opening and closing style tags in <style>code</style>, but leave the style code to confuse text parsing. This simplistic tag removal also causes words on either side of the tags to be joined, creating hard to parse text.

To fix these problems, you need to process certain tags first before using strip_tags(). This is easily done with a few regular expressions that:

• Remove HTML tag pairs and enclosed content for styles, scripts, embedded objects, etc.
• Add line breaks around block-level tags to prevent word joining problems after tag removal.

Once this is done, call strip_tags() to remove the remaining tags.

Below is sample code to do this. Its regular expressions are more verbose than strictly necessary, but it helps make the function clearer. More explanations follow in the sections after the code.

Downloads: strip_html_tags.zip.

/**
 * Remove HTML tags, including invisible text such as style and
 * script code, and embedded objects.  Add line breaks around
 * block-level tags to prevent word joining after tag removal.
 */
function strip_html_tags( $text )
{
    $text = preg_replace(
        array(
          // Remove invisible content
            '@<head[^>]*?>.*?</head>@siu',
            '@<style[^>]*?>.*?</style>@siu',
            '@<script[^>]*?.*?</script>@siu',
            '@<object[^>]*?.*?</object>@siu',
            '@<embed[^>]*?.*?</embed>@siu',
            '@<applet[^>]*?.*?</applet>@siu',
            '@<noframes[^>]*?.*?</noframes>@siu',
            '@<noscript[^>]*?.*?</noscript>@siu',
            '@<noembed[^>]*?.*?</noembed>@siu',
          // Add line breaks before and after blocks
            '@</?((address)|(blockquote)|(center)|(del))@iu',
            '@</?((div)|(h[1-9])|(ins)|(isindex)|(p)|(pre))@iu',
            '@</?((dir)|(dl)|(dt)|(dd)|(li)|(menu)|(ol)|(ul))@iu',
            '@</?((table)|(th)|(td)|(caption))@iu',
            '@</?((form)|(button)|(fieldset)|(legend)|(input))@iu',
            '@</?((label)|(select)|(optgroup)|(option)|(textarea))@iu',
            '@</?((frameset)|(frame)|(iframe))@iu',
        ),
        array(
            ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ',
            "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0",
            "\n\$0", "\n\$0",
        ),
        $text );
    return strip_tags( $text );
}

Example

Read an HTML file, convert to UTF-8, strip out HTML tags and invisible content, and decode HTML entities into UTF-8:

/* Read an HTML file */
$raw_text = file_get_contents( $filename );
 
/* Get the file's character encoding from a <meta> tag */
preg_match( '@<meta\s+http-equiv="Content-Type"\s+content="([\w/]+)(;\s+charset=([^\s"]+))?@i',
    $raw_Text, $matches );
$encoding = $matches[3];
 
/* Convert to UTF-8 before doing anything else */
$utf8_text = iconv( $encoding, "utf-8", $raw_text );
 
/* Strip HTML tags and invisible text */
$utf8_text = strip_html_tags( $utf8_text );
 
/* Decode HTML entities */
$utf8_text = html_entity_decode( $utf8_text, ENT_QUOTES, "UTF-8" ); 

Explanation

For search engine indexing or keyword analysis, you need to read an HTML web page and strip away everything but the raw text. This gives you a clean word list that you can tokenize, stem, analyze, and index.

PHP's standard strip_tags() function is widely used for removing tags, mostly because it is built into PHP. However, as I said earlier, it has two important limitations that have to be addressed:

1. It leaves behind text that should be invisible, such as style and script code.
2. It joins together words to the left and right of a removed tag.

Removing invisible text

Style, script, and other invisible page code is enclosed by tag pairs like <script>...</script>. A regular expression can easily find these. But be sure to use pattern modifiers to do case-insensitive matches (an "i" pattern modifier) and be UTF-8 safe (a "u" pattern modifier). This insures that your code will work on upper, lower, and mixed-case tags and on international web pages.

@<script[^>]*?>.*?</script>@siu

The sample code at the top of this article uses a similar expression for several tag pairs that surround code and invisible text:

• <head>...</head> encloses header information, such as the title and page styles and scripts.
• <style>...</style> encloses page styles.
• <script>...</script> encloses page scripts.
• <object>...</object> encloses an object and its parameters.
• <embed>...</embed> (deprecated) encloses a plug-in and its parameters.
• <applet>...</applet> (deprecated) encloses a Java applet and its parameters.
• <noframes>...</noframes> encloses text to show if frames are not supported.
• <noscript>...</noscript> encloses text to show if scripts are not supported.
• <noembed>...</noembed> (deprecated) encloses text to show if a plug-in cannot be embedded.

There are a few more tag pairs whose enclosed content you may want to remove:

• <area>...</area> encloses an area in a map.
• <map>...</map> encloses an area in an image map.
• <marquee>...</marquee> (deprecated) encloses text for a scrolling marquee.
• <menu>...</menu> (deprecated) can enclose options for a menu.
• <select>...</select> can enclose options for a drop-down menu.
• <textarea>...</textarea> encloses text to be shown within a multi-line text area.

Preventing incorrect word joins

When PHP's strip_tags() removes HTML tags, the text on either side of the tag is joined. For example, this text:

<p>First <strong>J</strong>oined paragraph</p><p>Second <strong>J</strong>oined paragraph</p>

becomes this:

First Joined paragraphSecond Joined paragraph

For inline tags like <strong> or <a>, word joining is the right behavior. The word "Joined" above is correctly created by joining the bold "J" with the normal "oined".

But for block tags like <p> or <div>, word joining is not correct. These tags have an implicit line break before and after the block. In the example above, the words "Paragraph" and "Second" should not have been joined.

To correct this problem, the sample code in this article recognizes block tags and inserts line breaks before and after each block.

Removing HTML tags

Finally, use PHP's strip_tags() to remove the remaining HTML tags and comments.

Other issues

For this to work reliably:

• Use the page's content type to get its character set, then convert to UTF-8 using the iconv() function. This insures that the text is in a known character encoding. The preg_replace() function handles multi-byte characters in UTF-8 if the /u pattern modifier is included.
• Strip HTML tags before decoding HTML entities. Decoding some entities, such as < and >, generates characters that will confuse tag parsing.
• HTML web pages must be syntactically correct (or reasonably close). For instance, if there is an open <script> tag and no closing tag, everything from the tag to the end of the file will be removed.

Downloads

• strip_html_tags.zip
  • Includes strip_html_tags.php. The code is covered by the OSI BSD license so you can use, modify, redistribute, and sell as you see fit.

Further reading

Related articles at NadeauSoftware.com

• PHP tip: How to get a web page using the fopen wrappers. Use PHP's file reading functions to get a web page, handling web server redirects and user-agent strings.
• PHP tip: How to get a web page using CURL. Use PHP's CURL (Client URL) functions to get a web page, handling web server redirects, compressed content, cookies, and user-agent strings.
• PHP tip: How to get a web page content type. Get the content type (and character set encoding) for a web page by parsing the HTTP header or looking for a <meta> tag in the file.
• PHP tip: How to decode HTML entities on a web page. Process a web page to convert all HTML entities (such as €) into UTF-8 multibyte characters.
• PHP tip: How to extract keywords from a web page. Get a good list of keywords from a web page by getting the web page text, converting it to UTF-8, stripping away HTML tags, punctuation, symbols, and numbers, and breaking the text into words.

Web articles and specifications

• HTML element. Wikipedia has a good summary of HTML and XHTML tags, grouping them into block and inline elements.