couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaakko Sipari (JIRA)" <j...@apache.org>
Subject [jira] [Created] (COUCHDB-1176) CouchDB accepts data which it cannot replicate (invalid UTF-8 json during replication)
Date Mon, 23 May 2011 13:31:47 GMT
CouchDB accepts data which it cannot replicate (invalid UTF-8 json during replication)
--------------------------------------------------------------------------------------

                 Key: COUCHDB-1176
                 URL: https://issues.apache.org/jira/browse/COUCHDB-1176
             Project: CouchDB
          Issue Type: Bug
    Affects Versions: 1.0.2, 1.0.1
         Environment: CentOS 5.5 64bit
            Reporter: Jaakko Sipari
            Priority: Critical
         Attachments: fffe_escaped.json, fffe_utf8.json

CouchDB appears to treat some unicode characters as illegal when parsing escaped unicode values
(\uXXXX) during insert or update of a document.  These characters can however be inserted
to the database by using the UTF-8 encoding instead of escaping. An example value would be
an unicode value 0xFFFE which is escaped \uFFFE and as UTF-8 is represented by consecutive
bytes with values 0xEF 0xBF and 0xBE.

Even though the documents are inserted using UTF-8 encoding without errors, couchdb always
serves them in the escaped form. This leads us to the actual problem we currently have. If
documents containing such unaccepted characters are inserted to couchdb by using UTF-8 encoding,
attempt to replicate the database will abort to first of those documents giving an error like
this:
{"error":"json_encode","reason":"{bad_term,{nocatch,{invalid_json,<<\"[{\\\"ok\\\":{\\\"_id\\\":\\\"192058c4f81afc66c5bf883548004331\\\",\\\"_rev\\\":\\\"1-ad1c9dcee520d12abdf948d91e31cf15\\\",\\\"abc\\\":\\\"\\\\ufffe\\\",\\\"_revisions\\\":{\\\"start\\\":1,\\\"ids\\\":[\\\"ad1c9dcee520d12abdf948d91e31cf15\\\"]}}}]\\n\">>}}}"}

Here are steps to reproduce:

curl -X PUT http://localhost:5984/replicationtest_source
curl -X PUT http://localhost:5984/replicationtest_target
# Should fail
curl -H "Content-Type:application/json" -X POST -d @fffe_escaped.json http://localhost:5984/replicationtest_source
# Should succeed
curl -H "Content-Type:application/json" -X POST -d @fffe_utf8.json http://localhost:5984/replicationtest_source
# Should fail to json_encode error related to the previously inserted document
curl -H "Content-Type:application/json" -X POST -d "{\"source\":\"http://localhost:5984/replicationtest_source\",\"target\":\"replicationtest_target\"}"
http://localhost:5984/_replicate

If anyone has a quick fix for this (how to accept "invalid" escaped unicode characters at
least during replication), we would be more than happy to test it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message