As mentioned by Ben Ramsey today, Maggie Nelson bumped into an issue in one of her applications with character sets and the incorrect storage/retrieval of information:

Even Oracle, which usually gets storing of data right on the money has had issues with character sets. [...] Needless to say, even when you *know* you set up your database correctly for supporting UTF8, the path to debug issues may be frustrating and full of red herrings.

She mentions the setup the application is using (NLS_CHARACTERSET AL32UTF8, NLS_NCHAR_CHARACTERSET AL16UTF16) but something wasn't right. The problem popped up when they tried to store Chinese characters into the database with the result of invalid data on a select.

After following several different leads, they finally came upon the culprit - the Apache process didn't have the access it needed to a directory in the ORACLE_HOME. In the end, it all only broke down into three easy steps to fix a very frustrating issue.