couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Tail Append Headers
Date Wed, 20 May 2009 16:48:17 GMT
On May 20, 2009, at 11:34 AM, Damien Katz wrote:

>
> On May 20, 2009, at 11:26 AM, Paul Davis wrote:
>
>> On Wed, May 20, 2009 at 11:22 AM, Damien Katz <damien@apache.org>  
>> wrote:
>>>
>>> On May 20, 2009, at 11:09 AM, Damien Katz wrote:
>>>>
>>>> Previously, only btree nodes were saved compressed and docs were  
>>>> not. I
>>>> didn't realize the compression was so expensive, but now that I  
>>>> switch it
>>>> off on both the branch and on trunk, I see big performance boosts  
>>>> for both.
>>>> And now the tail append stuff is slightly faster on my machine.
>>>
>>> To clarify, disabling the compression completely on both trunk and  
>>> the
>>> branch results in big performance increases for both, with the  
>>> tail_header
>>> branch now being slightly faster than trunk running the lightning  
>>> test on my
>>> machine.
>>>
>>> -Damien
>>>
>>
>> Awesome. Is there a noticeable size difference on the database files?
>
> It looks to take about 2x as much diskspace as without compression.

Nice find.  I also see the the tail_header branch slightly faster than  
trunk with compression turned off on both, and the DB size increased  
by ~2x.  For kicks I tried turning the compression level down to 1  
(default is 6 on a 1-9 scale).  Running hovercraft:lightning() gives me

compression level   insert rate     db size
0                   11725           16.7MB
1                    4186            8.2MB
6 (default)          3938            7.8MB

So it's still a huge cost.  The nice thing is that binary_to_term  
seems perfectly happy reading a mix of compressed and uncompressed  
binaries, which means the compression level can be a configuration  
parameter if we want it to be.  gzip decompresses pretty quickly, so  
I'm guessing that reading a compressed DB will be faster than an  
uncompressed one.  We'll have to measure it, though.

Adam

Mime
View raw message