couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Anderson <jch...@apache.org>
Subject Re: view size extremely disk inefficient
Date Tue, 07 Jul 2009 21:04:19 GMT
Peter,

I'd totally suggest trying your code on trunk. We'd be eager to hear
about any performance changes you notice as we're starting on the
0.10.0 release. The file writer is much more efficient, etc.

Most things should work and you ought to be able to upgrade databases
to the new format with compaction. (Keep backups!)

Chris

On Tue, Jul 7, 2009 at 1:52 PM, Peter Hsu<phsu@abaca.com> wrote:
> Is that available in 0.9.0?
>
> I tried a POST but I get a 405 error.
>
> [Tue, 07 Jul 2009 13:46:08 GMT] [info] [<0.22689.306>] 172.16.80.64 - - 'GET' /mydb/_compact/meta_in
404
>
> [Tue, 07 Jul 2009 13:47:43 GMT] [info] [<0.22706.306>] 172.16.80.64 - - 'POST'
/mydb/_compact/meta_in 405
>
> -----Original Message-----
> From: Adam Kocoloski [mailto:kocolosk@apache.org]
> Sent: Tuesday, July 07, 2009 12:16 PM
> To: user@couchdb.apache.org
> Subject: Re: view size extremely disk inefficient
>
> On Jul 7, 2009, at 2:56 PM, Peter Hsu wrote:
>
>> I need help explaining why the sizes of my views are so large.
>>
>> The emitted rows for the document have a key length of about 40
>> bytes (it's an array, if that matters) and a view length of about
>> 400 bytes (raw json).   However, I'm seeing over 2k/row average over
>> the view.  Factoring in the overhead of writing the btrees still
>> doesn't really make sense.
>>
>> At 10M docs, I have almost 3.2k/message.  The views were generated
>> at 3000 doc increments.  At the beginning, with the first view
>> generation, the view size was about 2MB.  This is about 800 bytes
>> per row, which could be reasonable.
>>
>> I measured the incremental size of the view after every 3000 rows
>> were added to the view.  By the time I'm at 30k rows, I'm seeing an
>> increment in the view size of 4MB, which is over 1k/doc.  By the end
>> of 10M messages, it's over a 10MB increment, which is over 3k/doc.
>>
>> It may be interesting that a lot of my keys are identical.  Does
>> that affect things?
>>
>> This is with the 0.9.0 running on cent5 (64 bit).
>>
>> Peter
>
> Hi Peter, have you tried compacting the view?
>
> POST /dbname/_compact/designname
>
> Best, Adam
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Mime
View raw message