I would rather not have to convert these weird characters to the HTML character entities, if possible. I'd rather be able to use these characters directly on the web page. If this is for some reason a bad idea, let me know. This might be more of a general web design question (i already posted it there), but I figured it is still appropriate to post here as well since PHP is used to pull an entry from the database, and I figured a lot of you here would know the answer to the question.

The general consensus is to use UTF8 in this case, but there's a few reminders for the poster too:

• Don't forget to make the database UTF8 too
• Be sure you're sending the right Content-Type for the UTF8 data
• an link to an article about what "developers must know about unicode/charactersets"

One set of changes that I think is particularly important was largely overlooked: For PHP 5.4 cataphract (Artefacto on StackOverflow) heroically rewrote large parts of htmlspecialchars thus fixing various quirks and adding some really nice new features. Here a quick summary of the most important changes: UTF-8 as the default charset, improved error handling (ENT_SUBSTITUTE) and Doctype handling (ENT_HTML401,...).

He goes into each of these three main features in a bit more detail, providing code to illustrate the improved error handling and the new flags for Doctype handling (covering HTML 4.01, HTML 5, XML 1 and XHTML).

The PatchworkUtf8 class implements the quasi complete set of string functions that need UTF-8 grapheme clusters awareness. These functions are all static methods of the PatchworkUtf8 class. The best way to use them is to add a use PatchworkUtf8 as u; at the beginning of your files, then when UTF-8 awareness is required, prefix by u:: when calling them.

In the README for the tool he talks about the functions included in the current release that match PHP's string functions as well as some additional methods like "isUtf8", "bestFit" and "strtocasefold". It relies on the mbstring, iconv and intl extensions being installed, and if they aren't, it falls back to other functionality (list of those methods included).

I had a problem last week with converting UTF-16 encoded strings to UTF-8 using PHP's iconv library on a Linux server. my code worked fine on my machine but the same code resulted in a rubbish unreadable characters on our production server.

In his example (a basic "Hello World" in Arabic) he notes that there's no byte order mark on the string and, because of this, the iconv feature tries to guess if it's big-endian or little-endian. This guessing varies from machine to machine resulting in the inconsistencies he saw. The solution is to define the "to" and "from" for the conversion manually rather than letting it just guess.

One such thing is the occurrence of the UTF byte order mark, or BOM. [...] For UTF-8, especially on Windows, it has become more and more common to use it to indicate that the file is indeed UTF. Most text editors handle this well and you won't ever see these bytes. As it should be.

He points out what could cause an issue - using strcmp or substr but it can be prevented by looking at and removing those first three bytes if needed. He includes a snippet of code that does just that.

It turns out basename does a bit more than just splicing the string at the last slash, because it's locale aware. In my case I was dealing with a multi-byte UTF-8 string. It took me quite some time figuring out what was going on, because I was testing from the console which had the en_US.UTF-8 locale, and the bug was appearing on Apache, which defaults to the C locale.

He includes an example snippet of code showing how it can work with both the default (well, for Apache anyway) of the "C" locale versus the "UTF-8" locale and return different results for the same urldecoded information.

UTF8 allows your site to represent characters other than those in the basic english alphabet. More often than not, your CodeIgniter Application will contain methods where users can enter their name. [...] This guide assumes you are reasonably competent in installing php extensions, adding config variables to your php.ini, and using MY_ CodeIgniter overloading. If you're not sure about any of these, please make sure you consult a professional.

You'll need to install the mbstring extension for PHP to be able to follow along with his example. He shows how to override the basic form functionality with custom functions to change the display of the form and how it handles the submitted information. He also looks at how to update the XML-RPC library that comes with the framework and the creation of a new helper to allow you to convert, check, compare and sort UTF-8 data.

Writing an application is easy. Writing an application that supports all characters from multiple languages? Not so easy. [...] To make your CodeIgniter application play nicely with UTF-8 you have a few things to think about.

He shows how to set the headers up correctly, change the framework's configuration to include UTF-8 as a character set and set up the database connection to use it too. If you're already using a non-UTF-8 database structure, he also includes an example of how to make the conversion.

While I was looking at some AJAX calls, I started to have a problem, for some reason, when I tried to query a JSON service I did using JQuery, the result was null for some fields. Going a little deeper, I notice that the records from the DB were OK, and the JavaScript was OK to, so what was the problem? The JSON Encode!

His issue stemmed from the character encoding of the string being passed into json_encode (in this case, UTF-8) was coming from a PHP script saved in a page encoded as ISO-8859-1. You can download the files he's come up with to illustrate the point.

Everyone else probably already knows this stuff, but I hit an issue today to that took a while to sort out. Fortunately, some kind folks on IRC helped me, but as it's embarrassing to ask for help on the same issue twice, I'm writing down what I've learned! The problem: Get a £ character stored to MySQL, retrieved and then displayed without any weird characters in front of it using UTF8.

His solution? Make sure you're using UTF-8 everywhere, not just when trying to insert into the database - in the broser's headers (both going in and coming out) and in the MySQL database insert. He gives code examples for each including database examples for PDO and the Zend_Db component of the Zend Framework.