incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Genereux <>
Subject Re: invalid UTF-8 JSON
Date Fri, 20 Aug 2010 05:46:22 GMT
Landon,  I had a nightmare a while back and posted the scenario and
solution on here.  I wasn't dealing with Chef but Rails was very happy
exporting invalid JSON from a character set perspective.  Here it is
again and hope it helps!

--- Problem ---
I'm exporting a portion of a massive MySQL database that I'm thinking
is better suited to a CouchDB database.  The MySQL database supports a
Rails application and the MySQL server is set to UTF8 for everything.
 I'm using to_json in Rails that appears to convert the records just
fine to JSON.  I get about 200 records converted and imported into
CouchDB and then the process dies with "Invalid UTF-8 JSON".  One of
my fields in the offending record has the word "fĂȘte".  The JSON
produced by Rails doesn't convert this UTF-8 character to the JSON \u0000
notation.  I don't think it should have to but maybe I'm not
understanding the standard.

--- Solution ---
The original importer of the data took ISO-8599-1 data and jammed it
into a UTF-8 field in the database.  The character that I was having
problems with was being auto translated when viewed in phpMyAdmin by
the web browser using a kind of ASCII/IOS-8859-1/Windows-1252 fallback
on non UTF-8 characters.  So the character that looked fine was actual
not UTF-8. I could cut and paste the converted Unicode character from
phpMyAdmin right into Futon and the JSON was valid.  The solution
within Rails was after I converted the Rails object to JSON, I ran
"json_data = Iconv.conv( 'utf-8', 'iso-8859-1', json_data )" to clean
out bad characters.  Worked like a charm!

On Thu, Aug 19, 2010 at 8:31 PM, Landon Clark <> wrote:
> Response inline
> On Thu, Aug 19, 2010 at 4:45 PM, Randall Leeds <> wrote:
>> If you recently updated couch be sure you don't have an old version of
>> the mochiweb libraries around anywhere.
>> See for example:
> Good hint. I double checked this and the only version that I have is:
> ...../lib/couchdb/erlang/lib/mochiweb-r113
> It appears that's an ok version right?
> Thanks,
> Landon

View raw message