couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Anderson <jch...@apache.org>
Subject Re: Timeout Error when trying to access views + Indexing problems
Date Sun, 04 Oct 2009 06:40:28 GMT
On Sat, Oct 3, 2009 at 11:27 PM, Glenn Rempe <glenn@rempe.us> wrote:
> And just another data point.  I was seeing timeouts from my CouchRest client
> when trying to do bulk saves to this large DB from my migration script.
> You can see that I find a batch of 1000 records to migrate and bulk save
> them to CouchDB.  I write back a timestamp to MySQL for each of those
> records.  So you can see it takes less than 10 seconds to update 1000 docs
> in mysql, and almost a minute for the http request to return 201 success for
> the bulk insert of that date into couchdb.  Avg record size is a few hundred
> bytes.
>
> I ended up having to fork mattetti CouchRest so that I could set an explicit
> timeout for the embedded RestClient to 120 seconds (!) in order for it to
> not barf...  Is it expected that larger DB's will not take constant time to
> add a number of records on in a bulk insert?

The next remedy here is to try one of the new uuid formats available in 0.10

[uuids]
algorithm = sequential

in your local ini should do it.

this will help the btree to require less disk seeks to load the path
to each updated node, as sequential ids are more likely to be in FS
cache.

I'm assuming your bulk saves are mostly new docs.

Chris


>
> Here is the output of some crude logging.
>
> 2009-10-04 06:16:55.854220 sync-mysql-to-couchdb(4241) [INFO]
> sync-mysql-to-couchdb-daemon.rb:20: Found 1000 records to try and migrate.
>
> [ I updated each mysql records timestamp individually in this gap.  About 9
> seconds. ]
>
> 2009-10-04 06:17:04.876943 sync-mysql-to-couchdb(4241) [INFO]
> sync-mysql-to-couchdb-daemon.rb:92: Now trying to save 1000 records to
> CouchDB out of 1000 found in MySQL.
>
> [This is the bulk post to CouchDB in this gap.  57 seconds!]
>
> 2009-10-04 06:18:01.989428 sync-mysql-to-couchdb(4241) [INFO]
> sync-mysql-to-couchdb-daemon.rb:94: Bulk save of 1000 of records to CouchDB
> complete.
>
> Would you suspect couchdb?  Or possibly IO probs with Elastic Block Store
> (EBS)?
>
> G
>
> On Sat, Oct 3, 2009 at 11:07 PM, Glenn Rempe <glenn@rempe.us> wrote:
>
>> Thanks for the reply Paul.  Some comments below.
>>
>> Also, just for full disclosure, the CouchDB I am working on was moved out
>> of another couchdb and it was originally created using CDB 0.9.1.  I show a
>> dir listing below that indicates exactly what was moved.
>>
>> On Sat, Oct 3, 2009 at 6:46 PM, Paul Joseph Davis <
>> paul.joseph.davis@gmail.com> wrote:
>>
>>> Glenn,
>>>
>>> This sounds like your map function is timing out which causes the error.
>>> You could try upping the os process timeout setting in the config.
>>>
>>>
>> When I go into futon and select one of my views in my design doc it
>> *always* consistently pops up a javascript alert with the error text at ~5
>> seconds after selecting the view.  It doesn't seem to matter what else I do.
>>  It also didn't vary when I changed the os_process_timeout value in futon as
>> you suggested from 5000 to 25000.  Can you explain exactly what this
>> particular param is doing?  I assume the value is milliseconds?
>>
>>
>>> To see what's going on you can increase to debug logging or use the log
>>> function in your maps. There's also the status page in futon which I think
>>> you said you were looking at.
>>>
>>>
>> Yes, I was previously looking at the status page.  But now since I've
>> upgraded to trunk I never see any indexing activity happening in the status
>> page no matter what I do.
>>
>>
>>> If indexing crashes it should just pick up where it left off when you
>>> retrigger. Use the status page to verify. If it's not then let us know.
>>>
>>>
>> Can you clarify, is this also the case when no index has ever successfully
>> run?  I was wondering if I first need to get through at least one index
>> session (maybe with a smaller amount of records) prior to incremental
>> indexing working as expected.
>>
>> Is there any way to determine what percentage of the total records have
>> been added to the index?
>>
>> For your info, here are the contents of the DB dir.  You can see the main
>> DB is 42GB now (~17 million records).
>>
>> root@ip-10-250-55-239:/home/rails/underscore-sync-mysql-to-couchdb/shared/log#
>> ls -la /vol/couchdb/var/lib/couchdb
>> total 41674956
>> drwxr-xr-x 3 couchdb root             78 2009-10-04 00:46 .
>> drwxr-xr-x 3 couchdb root             20 2009-10-03 05:02 ..
>> -rw-r--r-- 1 couchdb couchdb 42675073133 2009-10-04 02:13
>> searchlight_production.couch
>> drwxr-xr-x 2 couchdb couchdb         120 2009-10-03 06:02
>> .searchlight_production_design
>>
>> root@ip-10-250-55-239:/home/rails/underscore-sync-mysql-to-couchdb/shared/log#
>> ls -la /vol/couchdb/var/lib/couchdb/.searchlight_production_design/
>> total 33700196
>> drwxr-xr-x 2 couchdb couchdb         120 2009-10-03 06:02 .
>> drwxr-xr-x 3 couchdb root             78 2009-10-04 00:46 ..
>> -rw-r--r-- 1 couchdb couchdb  9819347287 2009-10-03 08:04
>> 5f190ffb2ed589746e8796d2423869ac.view
>> -rw-r--r-- 1 couchdb couchdb    91402872 2009-10-03 06:03
>> b127a58306fb8e7858cd1a92f8398511.view
>> -rw-r--r-- 1 couchdb couchdb 24598236884 2009-10-02 13:00
>> SearchDocument.view
>>
>>
>>
>>> If you can't find anything in the debug logs then ping the lust and we'll
>>> get into trying to duplicate.
>>>
>>>
>> I have turned on the 'debug' level in the logs and that provided me with
>> the info I previously provided.  I'll try to use the log function in the map
>> and see if that shows anything.
>>
>> Thanks for helping.  If it comes to it, I may be able to make a snapshot of
>> this EBS volume and start a host that you could login to and get your hands
>> directly on it if that would be helpful.
>>
>> Glenn
>>
>>
>>
>
>
> --
> Glenn Rempe
>
> email                 : glenn@rempe.us
> voice                 : (415) 894-5366 or (415)-89G-LENN
> twitter                : @grempe
> contact info        : http://www.rempe.us/contact.html
> pgp                    : http://www.rempe.us/gnupg.txt
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Mime
View raw message