couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filipe Manana (JIRA)" <j...@apache.org>
Subject [jira] Commented: (COUCHDB-1009) Make couch_stream buffer configurable
Date Mon, 03 Jan 2011 14:38:45 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976759#action_12976759
] 

Filipe Manana commented on COUCHDB-1009:
----------------------------------------

Just by increasing the buffer size from 4Kb to 64Kb, for the same attachment upload, I got
a reduction from 1545051 us to 1206335 us.

It's not that much, but I believe this stresses less the file driver. Writing larger chunks
is usually better than writing many small chunks (reduces number of context switches, better
chances of having contiguous blocks allocated by the filesystem). Plus, after file:allocate/2
lands in OTP, it will be better to use it with even larger sizes (512Kb, 1Mb for example).

For e.g., running the test function at https://github.com/fdmanana/couchdb/commit/6ee38e330aa5bff51d58470cba1096f01aa0dfbd#L2R39

Erlang R14B01 (erts-5.8.2) [source] [smp:2:2] [rq:2] [async-threads:4] [hipe] [kernel-poll:true]

Eshell V5.8.2  (abort with ^G)
1> Apache CouchDB 1.2.0a13d6dd1-git (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.37.0>] Apache CouchDB has started on http://127.0.0.1:5984/

1> couch_file:test(4096, 16).
multi writes of 16 binaries, each of size 4096 bytes, took 870us
batch write of 16 binaries, each of size 4096 bytes,  took 377us
ok
2> couch_file:test(4096, 16).
multi writes of 16 binaries, each of size 4096 bytes, took 1271us
batch write of 16 binaries, each of size 4096 bytes,  took 366us


> Make couch_stream buffer configurable
> -------------------------------------
>
>                 Key: COUCHDB-1009
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1009
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>         Environment: trunk
>            Reporter: Filipe Manana
>            Assignee: Filipe Manana
>            Priority: Trivial
>         Attachments: COUCHDB-1009.patch
>
>
> The couch_stream buffer is hardcoded to 4Kb.
> This value should be configurable. Larger values can improve write and specially read
performance (if we write larger chunks to disk, we have higher chances of reading more contiguous
disk blocks afterwards). 
> I also think it's a good idea to change the default value from 4Kb to something higher
(64Kb for e.g.).
> Patch attached

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message