Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 32548 invoked from network); 1 Jul 2009 23:57:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Jul 2009 23:57:57 -0000 Received: (qmail 75671 invoked by uid 500); 1 Jul 2009 23:58:06 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 75596 invoked by uid 500); 1 Jul 2009 23:58:06 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 75586 invoked by uid 99); 1 Jul 2009 23:58:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 23:58:06 +0000 X-ASF-Spam-Status: No, hits=1.7 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.68.5.17] (HELO relay03.pair.com) (209.68.5.17) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 01 Jul 2009 23:57:57 +0000 Received: (qmail 84045 invoked from network); 1 Jul 2009 23:57:34 -0000 Received: from 75.143.234.216 (HELO ?192.168.1.197?) (75.143.234.216) by relay03.pair.com with SMTP; 1 Jul 2009 23:57:34 -0000 X-pair-Authenticated: 75.143.234.216 Message-Id: <490BFA26-03EF-4BD9-806D-A3F4F2A76C59@apache.org> From: Damien Katz To: user@couchdb.apache.org In-Reply-To: <4A4BBB33.20001@borwankar.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Subject: Re: chunked encoding problem ? - error messages from curl as well as lucene Date: Wed, 1 Jul 2009 19:57:34 -0400 References: <921000906292111x5ca5cbbbh9b7b3aa7e9d123dc@mail.gmail.com> <921000906292115i77cbb18fhf0a5ab87681a53f0@mail.gmail.com> <224A41AE-3A54-495B-A810-805C057FBE95@apache.org> <921000906301118y3bb11937hf6c0cd433387e05d@mail.gmail.com> <4A4BBB33.20001@borwankar.com> X-Mailer: Apple Mail (2.935.3) X-Virus-Checked: Checked by ClamAV on apache.org How where you doing the uploads? Via individual PUTs or, by bulk request(s)? Because if via PUTs, it's might be URL parsing that needs validation. -Damien On Jul 1, 2009, at 3:38 PM, Nitin Borwankar wrote: > Chris Anderson wrote: >> [...] >> Currently CouchDB doesn't validate json string contents on input, >> only >> on output. >> > > That seems problematic & inconsistent - if you let it in you should > at least let it be read. > In my case I uploaded a ton of stuff, saw no errors and then a huge > barf when doing .../_all_docs - with unhelpful error messages about > chunked encoding. > > Can I request more useful error messages when you detect an encoding > problem, if you decide need to keep the current read/write behavior. > > Nitin > >> Adding an option to block invalid unicode input would be a small >> patch, but perhaps slow things down as we'd have to spend more time >> in >> the encoder while writing. Worth measuring I suppose. >> >> Is this something users are running into a lot? I've heard this once >> before, if lots of people are seeing this, it's definitely worthy of >> fixing. >> >> I uploaded via Python httplib - not >> >>> couchdb-python. Is this a bug - the one that is fixed in 0.9.1? >>> >>> Nitin >>> >>> 37% of all statistics are made up on the spot >>> ------------------------------------------------------------------------------------- >>> Nitin Borwankar >>> nborwankar@gmail.com >>> >>> >>> On Tue, Jun 30, 2009 at 8:58 AM, Damien Katz >>> wrote: >>> >>> >>>> This might be the json encoding issue that Adam fixed. >>>> >>>> The 0.9.x branch, which is soon to be 0.9.1, fixes that issue. >>>> Try building >>>> and installing from the branch and see if that fixes the problem: >>>> svn co http://svn.apache.org/repos/asf/couchdb/branches/0.9.x/ >>>> >>>> -Damien >>>> >>>> >>>> >>>> On Jun 30, 2009, at 12:15 AM, Nitin Borwankar wrote: >>>> >>>> Oh and when I use Futon and try to browse the docs around where >>>> curl >>>> >>>>> gives >>>>> an error, when I hit the page containing the records around the >>>>> error >>>>> Futon >>>>> just spins and doesn't render the page. >>>>> >>>>> Data corruption? >>>>> >>>>> Nitin >>>>> >>>>> 37% of all statistics are made up on the spot >>>>> >>>>> ------------------------------------------------------------------------------------- >>>>> Nitin Borwankar >>>>> nborwankar@gmail.com >>>>> >>>>> >>>>> On Mon, Jun 29, 2009 at 9:11 PM, Nitin Borwankar >>>> >>>>>> wrote: >>>>>> >>>>> >>>>>> Hi, >>>>>> >>>>>> I uploaded about 11K + docs total 230MB or so of data to a 0.9 >>>>>> instance >>>>>> on >>>>>> Ubuntu. >>>>>> Db name is 'plist' >>>>>> >>>>>> curl http://localhost:5984/plist gives >>>>>> >>>>>> >>>>>> >>>>>> {"db_name":"plist","doc_count":11036,"doc_del_count": >>>>>> 0,"update_seq":11036,"purge_seq":0, >>>>>> >>>>>> >>>>>> "compact_running":false,"disk_size": >>>>>> 243325178,"instance_start_time":"1246228896723181"} >>>>>> >>>>>> suggesting a non-corrupt db >>>>>> >>>>>> curl http://localhost:5984/plist/_all_docs gives >>>>>> >>>>>> {"id":"adnanmoh","key":"adnanmoh","value":{"rev":"1-663736558"}}, >>>>>> >>>>>> >>>>>> {"id":"adnen.chockri","key":"adnen.chockri","value": >>>>>> {"rev":"1-1209124545"}}, >>>>>> curl: (56) Received problem 2 in the chunky >>>>>> parser <<--------- >>>>>> note curl >>>>>> error >>>>>> {"id":"ado.adamu","key":"ado.adamu","value": >>>>>> {"rev":"1-4226951654"}} >>>>>> >>>>>> suggesting a chunked data transfer error >>>>>> >>>>>> >>>>>> couchdb-lucene error message in couchdb.stderr reads >>>>>> >>>>>> [...] >>>>>> >>>>>> [couchdb-lucene] INFO Indexing plist from scratch. >>>>>> [couchdb-lucene] ERROR Error updating index. >>>>>> java.io.IOException: CRLF expected at end of chunk: 83/101 >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient.ChunkedInputStream.readCRLF(ChunkedInputStream.java: >>>>>> 207) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient >>>>>> .ChunkedInputStream.nextChunk(ChunkedInputStream.java:219) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient.ChunkedInputStream.read(ChunkedInputStream.java:196) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient >>>>>> .ChunkedInputStream.exhaustInputStream(ChunkedInputStream.java: >>>>>> 369) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient.ChunkedInputStream.close(ChunkedInputStream.java:346) >>>>>> at java.io.FilterInputStream.close(FilterInputStream.java:159) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient >>>>>> .AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java: >>>>>> 194) >>>>>> at >>>>>> >>>>>> org >>>>>> .apache >>>>>> .commons >>>>>> .httpclient >>>>>> .AutoCloseInputStream.close(AutoCloseInputStream.java:158) >>>>>> at >>>>>> com >>>>>> .github.rnewson.couchdb.lucene.Database.execute(Database.java: >>>>>> 141) >>>>>> at >>>>>> com.github.rnewson.couchdb.lucene.Database.get(Database.java:107) >>>>>> at >>>>>> >>>>>> com >>>>>> .github >>>>>> .rnewson.couchdb.lucene.Database.getAllDocsBySeq(Database.java: >>>>>> 82) >>>>>> at >>>>>> >>>>>> com.github.rnewson.couchdb.lucene.Index >>>>>> $Indexer.updateDatabase(Index.java:229) >>>>>> at >>>>>> >>>>>> com.github.rnewson.couchdb.lucene.Index >>>>>> $Indexer.updateIndex(Index.java:178) >>>>>> at com.github.rnewson.couchdb.lucene.Index >>>>>> $Indexer.run(Index.java:90) >>>>>> at java.lang.Thread.run(Thread.java:595) >>>>>> >>>>>> >>>>>> suggesting a chunking problem again. >>>>>> >>>>>> Who is creating this problem - my data? CouchDB chunking ? >>>>>> >>>>>> Help? >>>>>> >>>>>> >>>>>> >>>>>> 37% of all statistics are made up on the spot >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------------- >>>>>> Nitin Borwankar >>>>>> nborwankar@gmail.com >>>>>> >>>>>> >>>>>> >> >> >> >> >