Character encodings, especially when dealing with XML, in PHP can be a pain to say the least. Matthew Weir O'Phinney found this out first-hand when a script he was working with had a mixed character set in one of its strings, giving the XML parser in the SimpleXML functionality problems.

The discovery was to detect the encoding of the string itself (not really the content) and convert eveything in it to that encoding. How, you might ask? With the handy mb_detect_encoding and mb_convert_encoding functions. Of course, this functionality has to be compiled into PHP, but it's well worth it if it's exactly what you need.

Character encodings, especially when dealing with XML, in PHP can be a pain to say the least. Matthew Weir O'Phinney found this out first-hand when a script he was working with had a mixed character set in one of its strings, giving the XML parser in the SimpleXML functionality problems.

The discovery was to detect the encoding of the string itself (not really the content) and convert eveything in it to that encoding. How, you might ask? With the handy mb_detect_encoding and mb_convert_encoding functions. Of course, this functionality has to be compiled into PHP, but it's well worth it if it's exactly what you need.

David has posted the results from some of his testing with the Unicode support that's available in PHP, specifically in the context of XML.

A couple of weeks ago, Tim Bray posted about PHP and received a firestorm of comments.

As Tim updated his posting with comments, he linked to a two-year-old posting by Steve Minutillo about PHP4's inability to detect character encodings in XML files and other Unicode bugs. That caught me by surprise — after all, PHP uses the venerable Expat as its XML parsing engine (the same engine used in most programming environments other than Java), and if Expat wasn’t getting things right, then the PHP people must have gone way out of their way to misconfigure it.

So, in the rest of the post, he sets about testing these results for himself, using PHP version 4.4.0 and 5.0.5 to test with. He shows the code that he used to create the tests - it produces UTF-8 encoded text regardless of what type it's outputted as. There were some issues that he ran up against, but some of that it just due to the large abiguity that XML creation/handling has in PHP.

David has posted the results from some of his testing with the Unicode support that's available in PHP, specifically in the context of XML.

A couple of weeks ago, Tim Bray posted about PHP and received a firestorm of comments.

As Tim updated his posting with comments, he linked to a two-year-old posting by Steve Minutillo about PHP4's inability to detect character encodings in XML files and other Unicode bugs. That caught me by surprise — after all, PHP uses the venerable Expat as its XML parsing engine (the same engine used in most programming environments other than Java), and if Expat wasn’t getting things right, then the PHP people must have gone way out of their way to misconfigure it.

So, in the rest of the post, he sets about testing these results for himself, using PHP version 4.4.0 and 5.0.5 to test with. He shows the code that he used to create the tests - it produces UTF-8 encoded text regardless of what type it's outputted as. There were some issues that he ran up against, but some of that it just due to the large abiguity that XML creation/handling has in PHP.

In this post from the SitePoint PHP Blog, Harry Fuecks talks about a new package of software he's worked up to make it possible for PHP to handle UTF-8 encoded strings - PHP UTF-8.

Been messing around with bits of this code for a long time, in fact since first really getting to grips with Dokuwiki, but finally got the first release out.

PHP UTF-8 is intended to make it possible to handle UTF-8 encoded strings in PHP, without requiring the mbstring extension (although it uses mbstring if it's available). In short, it provides versions of PHP's string functions (pretty much everything you'll find on this list), prefixed with utf_ and aware of UTF-8 encoding (that 1character >= 1 byte). It also gives you some tools to help check UTF-8 strings for "well formedness", strip bad sequences and some "ASCII helpers".

He continues the post, mentioning where some of the code for it was pulled from and a note about the documentation (there, but scarce). He also includes a warning for the use of it - not to use it "blindly" and only to use it when you need it, not to replace the standard PHP str_* functions.

In this post from the SitePoint PHP Blog, Harry Fuecks talks about a new package of software he's worked up to make it possible for PHP to handle UTF-8 encoded strings - PHP UTF-8.

Been messing around with bits of this code for a long time, in fact since first really getting to grips with Dokuwiki, but finally got the first release out.

PHP UTF-8 is intended to make it possible to handle UTF-8 encoded strings in PHP, without requiring the mbstring extension (although it uses mbstring if it's available). In short, it provides versions of PHP's string functions (pretty much everything you'll find on this list), prefixed with utf_ and aware of UTF-8 encoding (that 1character >= 1 byte). It also gives you some tools to help check UTF-8 strings for "well formedness", strip bad sequences and some "ASCII helpers".

He continues the post, mentioning where some of the code for it was pulled from and a note about the documentation (there, but scarce). He also includes a warning for the use of it - not to use it "blindly" and only to use it when you need it, not to replace the standard PHP str_* functions.

In this new post on the SitePoint PHP Blog today, Harry looks at why it's "living dangerously" to use PHP with UTF-8.

Quick one—knocked up a list of "dangerous" functions and functionality in PHP, in relation to the use of UTF-8, available at http://www.phpwact.org/php/i18n/utf-8. These are for a "default" PHP setup without the mbstring overloading or PHP6 (where charset problems "magically vanish" ;) ).

This follows on from (unfinished) stuff here on charsets (tending towards UTF-8), which should help explain some of this.

He also notes that you can't rely on mbstring to be there, so he offers an alternative...

In this new post on the SitePoint PHP Blog today, Harry looks at why it's "living dangerously" to use PHP with UTF-8.

Quick one—knocked up a list of "dangerous" functions and functionality in PHP, in relation to the use of UTF-8, available at http://www.phpwact.org/php/i18n/utf-8. These are for a "default" PHP setup without the mbstring overloading or PHP6 (where charset problems "magically vanish" ;) ).

This follows on from (unfinished) stuff here on charsets (tending towards UTF-8), which should help explain some of this.

He also notes that you can't rely on mbstring to be there, so he offers an alternative...