couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Leeds (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COUCHDB-1334) Indexer speedup (for non-native view servers)
Date Tue, 08 Nov 2011 17:23:51 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146413#comment-13146413
] 

Randall Leeds commented on COUCHDB-1334:
----------------------------------------

Then again, Adam, we've got the SpiderMonkey issues to take care of and we still need to figure
out whether or not that changes 1.2 into 2.0 (unless maybe we have enough new stuff for 1.2
that we would just release a not-forward-compatible-with-SM-trunk major version anyway). We
should collectively make a decision about what to do with these branches soon.
                
> Indexer speedup (for non-native view servers)
> ---------------------------------------------
>
>                 Key: COUCHDB-1334
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1334
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core, JavaScript View Server, View Server Support
>            Reporter: Filipe Manana
>            Assignee: Filipe Manana
>             Fix For: 1.2
>
>         Attachments: 0001-More-efficient-view-updater-writes.patch, 0002-More-efficient-communication-with-the-view-server.patch,
master-0002-More-efficient-communication-with-the-view-server.patch
>
>
> The following 2 patches significantly improve view index generation/update time and reduce
CPU consumption.
> The first patch makes the view updater's batching more efficient, by ensuring each btree
bulk insertion adds/removes a minimum of N (=100) key/value pairts. This also makes the index
file size grow not so fast with old data (old btree nodes basically). This behaviour is already
done in master/trunk in the new indexer (by Paul Davis).
> The second patch maximizes the throughput with an external view server (such as couchjs).
Basically it makes the pipe (erlang port) communication between the Erlang VM (couch_os_process
basically) and the view server more efficient since the 2 sides spend less time block on reading
from the pipe.
> Here follow some benchmarks.
> test database at  http://fdmanana.iriscouch.com/test_db  (1 million documents)
> branch 1.2.x
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real	2m45.097s
> user	0m0.006s
> sys	0m0.007s
> view file size: 333Mb
> CPU usage:
> $ sar 1 60
> 22:27:20  %usr  %nice   %sys   %idle
> 22:27:21   38      0     12     50
> (....)
> 22:28:21   39      0     13     49
> Average:     39      0     13     47   
> branch 1.2.x + batch patch (first patch)
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real	2m12.736s
> user	0m0.006s
> sys	0m0.005s
> view file size 72Mb
> branch 1.2.x + batch patch + os_process patch
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real	1m9.330s
> user	0m0.006s
> sys	0m0.004s
> view file size:  72Mb
> CPU usage:
> $ sar 1 60
> 22:22:55  %usr  %nice   %sys   %idle
> 22:23:53   22      0      6     72
> (....)
> 22:23:55   22      0      6     72
> Average:     22      0      7     70   
> master/trunk
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real	1m57.296s
> user	0m0.006s
> sys	0m0.005s
> master/trunk + os_process patch
> $ echo 3 > /proc/sys/vm/drop_caches
> $ time curl http://localhost:5984/test_db/_design/test/_view/test1
> {"rows":[
> {"key":null,"value":1000000}
> ]}
> real	0m53.768s
> user	0m0.006s
> sys	0m0.006s

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message