couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Clark Hess <seanh...@gmail.com>
Subject Re: How to import data quickly
Date Thu, 28 Jan 2010 14:05:13 GMT
I tried both 1000 and 2000. I was getting timeouts from my http client
before (it had a 5 second timeout), and haven't tried taking it higher since
I changed the timeout. I'll try some values as see if I can figure out what
the best value is

On Thu, Jan 28, 2010 at 7:02 AM, Troy Kruthoff <tkruthoff@gmail.com> wrote:

> Just curious, what batch size did you use...  I was just getting to run
> some test data to see where the sweet spot is for our hardware, I remember
> reading somewhere that someone thought it was around 3k docs.
>
> Troy
>
>
>
> On Jan 28, 2010, at 4:21 AM, Sean Clark Hess wrote:
>
>  Sweet... down to 28 minutes with bulk. Thanks
>>
>> On Thu, Jan 28, 2010 at 4:25 AM, Sean Clark Hess <seanhess@gmail.com>
>> wrote:
>>
>>  Ah, I forgot about bulk! Thanks!
>>>
>>>
>>> On Thu, Jan 28, 2010 at 4:24 AM, Alex Koshelev <daevaorn@gmail.com>
>>> wrote:
>>>
>>>  How do you import data to CouchDB? Do you use _bulk API?
>>>> ---
>>>> Alex Koshelev
>>>>
>>>>
>>>> On Thu, Jan 28, 2010 at 1:51 PM, Sean Clark Hess <seanhess@gmail.com>
>>>> wrote:
>>>>
>>>>  I'm trying to import 7 million rows into couch from an xml document. If
>>>>>
>>>> I
>>>>
>>>>> use a database with a "normal" interface (comparing with Mongo here),
>>>>>
>>>> the
>>>>
>>>>> process completes in 37 minutes. If I use couch, it takes 10 hours. I
>>>>>
>>>> think
>>>>
>>>>> it might be due to the overhead of the http interface, but I'm not
>>>>> sure.
>>>>>
>>>>> Is there any way to get data in there faster?
>>>>>
>>>>> ~sean
>>>>>
>>>>>
>>>>
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message