incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Katz <dam...@apache.org>
Subject Re: Practical storage limit
Date Wed, 16 Jul 2008 23:02:07 GMT

On Jul 16, 2008, at 6:56 PM, David King wrote:

>> We'd love to hear what you come up with and also to solve any  
>> problems you might encounter on your way. Please let us know.  
>> Please note that CouchDB at this point is not optimised. We are  
>> still in the 'getting it right' phase before we come to the  
>> 'getting it fast'. That said, CouchDB is plenty fast already, but  
>> there is also the potential to greatly speed up things.
>
>
> So I'm trying a smaller version of this first (9 million records),  
> and I've hit a snag. I have some rather simple python code to read  
> from Postgres and write to couchdb (that uses couchdb-python, where  
> 'db' is a couchdb.client.Database object):
>
>    chunker = IteratorChunker(get_stuff())
>
>    while not chunker.done:
>        print "fetching"
>        chunk = chunker.next_chunk(1000)
>        if chunk:
>            print "Adding %d items, starting with %s" %  
> (len(chunk),chunk[0]['_id'])
>            db.update(chunk)
>
> db.update(docs) (see <http://code.google.com/p/couchdb-python/source/browse/trunk/couchdb/client.py

> >, line 360) uses the bulk API, like:
>
>    data = self.resource.post('_bulk_docs', content={'docs':  
> documents})
>
> At apparently random points throughout this process, but almost  
> always before 15,000 records or so, the process dies with an  
> exception, the tail end of which looks like:
>
>  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ 
> python2.5/httplib.py", line 707, in send
>    self.sock.sendall(str)
>  File "<string>", line 1, in sendall
> socket.error: (54, 'Connection reset by peer')
>
> If I have Futon up while it's running, I occasionally get a  
> Javascript error along the lines of "killed" (reproducing it is  
> difficult) at the same time.
>
> I could have it catch the reset connection and re-try, but why would  
> this be happening?
>

You appear to be hitting the weird mochiweb connection reset bug. It's  
causes test failures too. We are looking into it.


Mime
View raw message