incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cottlehuber <...@jsonified.com>
Subject Re: Replication of attachment is extremely slow
Date Fri, 24 Jan 2014 01:08:21 GMT
> On 23. Jänner 2014 at 23:28:11, Scott Weber (scotty2541@sbcglobal.net) wrote:
>
>From: Scott Weber <scotty2541@sbcglobal.net>
>To: "user@couchdb.apache.org" <user@couchdb.apache.org>
>Subject: Replication of attachment is extremely slow
> Date: 23. Jänner 2014 23:27:36 
> MEZ
I have seen the same thing as the  
> poster below.  Does anyone have a fix? Work around?
>
> is what I tried:
> I attached a mixed content file to a document  
> (text+binary, kind of like a PDF).  This was about 3 meg  
> or so. Then I triggered a replication to another database.   
> It took 30 seconds to replicate it *on the same box*.

> I deleted everything, and made several repeats of the this test.

Scott, Rian,

I don’t see this behaviour here. Would you mind trying a few things please?

- provide public accessible test files
- couch.log in debug mode preferably 
- get very specific on what is done, preferably via curl
- or python http tool from httpie.org***
- this ideally should be shell-scriptable
- and provide results incl timing
- any handwavey statements about your DB size & doc types, & overall couch instance
might be doing at that time?

e.g. I’m doing this, on a well-specd MBP early 2011, OSX 10.9.1,
running couch master 37c8459 on R16B03 in admin party, using 
3 files:

# approx 60MB pdf mainly with embedded images
https://github.com/steveklabnik/CLOSURE/blob/master/CLOSURE.pdf?raw=true
# 370K text-based PDF
http://tools.ietf.org/pdf/draft-ietf-ppsp-peer-protocol-08.pdf
# ~20MB pdf mainly text
http://hintjens.wdfiles.com/local--files/main%3Afiles/cc1pe.pdf

alias pie="http --verify=no --json --pretty all --verbose --style fruity”
COUCH=http://localhost:5984

# drop testy dbs and create new ones
pie delete $COUCH/testy
pie delete $COUCH/testy2
pie delete $COUCH/testy3
pie put $COUCH/testy
pie put $COUCH/testy2
pie put $COUCH/testy3

time pie put $COUCH/testy/ietf/ppsp-8 < /tmp/draft-ietf-ppsp-peer-protocol-08.pdf
>> .22s user 0.07s system 86% cpu 0.333 total

time pie put $COUCH/testy/cc1pe/cc1pe < /tmp/cc1pe.pdf
>> 0.22s user 0.11s system 0% cpu 2:31.16 total

time pie put $COUCH/testy/why/closure < /tmp/CLOSURE.pdf
>> 0.23s user 0.18s system 0% cpu 7:21.40 total

# replicate using _replicate API
time pie post $COUCH/_replicate source=testy target=testy2
>> 0.22s user 0.07s system 13% cpu 2.070 total

# replicate using _replicator DB
pie post $COUCH/_replicator _id=t3 source=testy target=testy3
pie get $COUCH/_replicator/t3
# its more work to get the duration out here, subtract the two once complete
>>  less than 2 seconds

For comparison

### The same files, to a remote large FreeBSD box running 1.5.0 + R16B02:

swift:
>> 0.26s user 0.09s system 26% cpu 1.311 total

cc1pe:
>>   0.31s user 0.50s system 0% cpu 1:46.56 total

closure:
>> 0.51s user 1.35s system 1% cpu 2:42.87 total

replication:
>> 0.27s user 0.09s system 14% cpu 2.477 total
>>  less than 2 seconds

### The same files to a local CouchDB 1.2.2 & R15B03:

swift:
>>   0.22s user 0.06s system 86% cpu 0.317 total

cc1pe:
>>  0.24s user 0.11s system 0% cpu 2:51.28 total

closure:
>> 0.23s user 0.20s system 0% cpu 4:55.81 total

replication:
>> 0.21s user 0.06s system 16% cpu 1.661 total

### ditto, but with cloudant on presumably linux shared infrastructure 

swift:
>> 0.26s user 0.08s system 25% cpu 1.342 total

cc1pe:
>> .79s user 0.54s system 3% cpu 42.128 total

closure:
>> 1.78s user 1.36s system 1% cpu 3:37.14 total

- skipped replication because its not a fair comparison.



*** I like httpie because I can do this:

   http post $COUCH/_replicate source=testy target=testy2

which is pretty damn handy.



Mime
View raw message