Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: what to do about invalid UTF-8 in saved documents?
From: Jan Lehnardt <jan@apache.org>
In-Reply-To: <AB5B796A-9DA4-423B-92BE-BE986897430C@apache.org>
Date: Wed, 1 Sep 2010 23:40:39 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <99A7FFAF-033B-4404-9ED4-2A3FE272B9BE@apache.org>
References: <AB5B796A-9DA4-423B-92BE-BE986897430C@apache.org>
To: dev@couchdb.apache.org

Thanks Adam for finding this one. I ran into it a couple of times and I =
thought I'm crazy.

I think the view server should skip the invalid doc and print a warning =
in the log file with the doc id when it does.

I believe a _bulk_doc request with a _deleted:true member still does =
allow removal of that doc, but I haven't tried in a while.

Cheers
Jan
--=20


On 31 Aug 2010, at 07:25, Adam Kocoloski wrote:

> It turns out that mochijson2 will incorrectly decode an invalid UTF-8 =
string if the illegal byte sequence in the string occurs after an =
escaped character (COUCHDB-875).  This means that one can store =
documents which will never be successfully retrieved or indexed in =
CouchDB 1.0.  Moreover, once one of these documents makes it into the DB =
a view build on that DB will never complete.
>=20
> I wonder what we should do to circumvent that problem?  At the very =
least it might make sense for the view indexer to skip documents which =
contain invalid UTF-8.
>=20
> Adam
>=20