Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Date: Tue, 28 Feb 2012 08:35:49 +0000 (UTC)
From: "Filipe Manana (Commented) (JIRA)" <jira@apache.org>
To: dev@couchdb.apache.org
Message-ID: 
 <483887585.27181.1330418149230.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1058168842.66776.1307126147519.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (COUCHDB-1186) Speedups in the view indexer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/COUCHDB-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218002#comment-13218002 ] 

Filipe Manana commented on COUCHDB-1186:
----------------------------------------

My replies in the following development mailing list thread:

http://mail-archives.apache.org/mod_mbox/couchdb-dev/201202.mbox/%3CCA%2BY%2B4475J_wPbiC%3Dg2R6CcqUfQ-_V6TTTxV2iS4xTbz9a10%2BXw%40mail.gmail.com%3E


> Speedups in the view indexer
> ----------------------------
>
>                 Key: COUCHDB-1186
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1186
>             Project: CouchDB
>          Issue Type: Improvement
>            Reporter: Filipe Manana
>            Assignee: Filipe Manana
>             Fix For: 1.2
>
>
> The patches at [1] and [2] do 2 distinct optimizations to the view indexer
> 1) Use a NIF to implement couch_view:less_json/2;
> 2) Multiple small optimizations to couch_view_updater - the main one is to decode the view server's JSON only in the updater's write process, avoiding 2 EJSON term copying phases (couch_os_process -> updater processes and writes work queue)
> [1] - https://github.com/fdmanana/couchdb/commit/3935a4a991abc32132c078e908dbc11925605602
> [2] - https://github.com/fdmanana/couchdb/commit/cce325378723c863f05cca2192ac7bd58eedde1c
> Using these 2 patches, I've seen significant improvements to view generation time. Here I present as example the databases at:
> A) http://fdmanana.couchone.com/indexer_test_2
> B) http://fdmanana.couchone.com/indexer_test_3
> ## Trunk
> ### database A
>         $ time curl http://localhost:5985/indexer_test_2/_design/test/_view/view1?limit=1
>         {"total_rows":1102400,"offset":0,"rows":[
>         {"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871       .73},{"x":153746.28,"y":190006.59}]}
>         ]}
>         real	19m46.007s
>         user	0m0.024s
>         sys	0m0.020s
> ### Database B
>         $ time curl http://localhost:5985/indexer_test_3/_design/test/_view/view1?limit=1
>         {"total_rows":1102400,"offset":0,"rows":[
>         {"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871       .73},{"x":153746.28,"y":190006.59}]}
>         ]}
>         real	21m41.958s
>         user	0m0.004s
>         sys	0m0.028s
> ## Trunk + the 2 patches
> ### Database A
>       $ time curl http://localhost:5984/indexer_test_2/_design/test/_view/view1?limit=1
>       {"total_rows":1102400,"offset":0,"rows":[
>       {"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871.7      3},{"x":153746.28,"y":190006.59}]}
>       ]}
>       real	16m1.820s
>       user	0m0.000s
>       sys	0m0.028s
>       (versus 19m46 with trunk)
> ### Database B
>       $ time curl http://localhost:5984/indexer_test_3/_design/test/_view/view1?limit=1
>       {"total_rows":1102400,"offset":0,"rows":[
>       {"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871.7      3},{"x":153746.28,"y":190006.59}]}
>       ]}
>       real	17m22.778s
>       user	0m0.020s
>       sys	0m0.016s
>       (versus 21m41s with trunk)
> Repeating these tests, always clearing my OS/fs cache before running them (via `echo 3 > /proc/sys/vm/drop_caches`), I always get about the same relative differences.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira