couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DA <daniel.adam...@gmail.com>
Subject Re: view/index rebuild in v2.0
Date Thu, 01 Dec 2016 13:17:49 GMT
Joan,
I only have measurements for the same database (v1.6.1) on a Linux with a
different HW: 4 x Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
10M docs, 10-view design doc rebuilt in 35hrs. Currently I'm running a
single-view rebuild, which is 1hr in, seems it could be finished within 8
hours.

For completeness, the view rebuild I initially wrote about (v2.0 on
Windows7) completed with the indexers for the respective shards taking from
32 to 42 hours. Our of curiosity, I will go on and test the SSD disk usage.

Thanks,
Daniel

>From    Joan Touzet <woh...@apache.org>
>Subject    Re: view/index rebuild in v2.0
>Date    Wed, 30 Nov 2016 18:57:08 GMT
>
>Please note that absolutely *zero* work has been done on our Windows
>port in terms of performance characterization and optimization.
>
>I'd be interested in hearing your results on the exact same hardware
>running the Linux version of CouchDB 2.0 - my hunch is that it will
>perform significantly better.
>
>-Joan
>
>----- Original Message -----
>> From: "DA" <daniel.adam903@gmail.com>
>> To: user@couchdb.apache.org
>> Sent: Wednesday, November 30, 2016 1:50:30 PM
>> Subject: Re: view/index rebuild in v2.0
>>
>> Hi,
>> thanks Peter, but I was thinking of a measures suitable for
>> production usage.
>>
>> Meanwhile, few observations:
>> - the 8 respective indexer tasks are not progressing evenly, some are
>> at 50%, some at 90%
>> - I noticed the system usage fell by almost 50%, both CPU and disk
>> activity.
>> So the system is now even more under-utilized, quite strange.
>>
>> When it's completed, I will try the same excercise on an SSD disk.
>>
>> Cheers, Daniel
>>
>>
>> On Wed, Nov 30, 2016 at 10:03 AM Peyton Vaughn <pvaughn@6fusion.com>
>> wrote:
>>
>> > Hi Daniel,
>> > Do you have 'delayed_commits' set to true? It's now defaulted to
>> > false in
>> > couch 2.0, but it really causes a pretty hefty performance hit when
>> > disabled.
>> >
>> > Peyton
>> >
>> > On Wed, Nov 30, 2016 at 5:57 AM, Daniel Adam
>> > <daniel.adam903@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > > I'm running an index/view re-generation with 10M documents,
>> > > 10views,
>> > having
>> > > 1 node and 8 shards on it.
>> > > It seems it will take more than 1 day. What are limiting factors
>> > > for
>> > that?
>> > > Looking at perfromance monitoring tools (Task Manager and Process
>> > > Explorer), it seems neither I/O or CPU is saturated:
>> > > CPU (8 cores) each core taking <30% its capacity
>> > > I/O reads ~ 8MB/s
>> > > I/O writes ~ 1.5MB/s
>> > >
>> > > Windows 7 64bit
>> > > 16GB memory
>> > > Intel i7-4800 @ 2.7GHz
>> > > HDD Seagate ST500LM0
>> > >
>> > > Any explanation why the system limits are not being saturated?
>> > > Seems to
>> > me
>> > > the bottleneck is I/O, but the disk is capable of 100MB/s
>> > > throughput in
>> > > sequential reads/writes. If the limiting factor is indeed I/O, I
>> > > assume
>> > > changing number of shards would not gain much, possibly even 1
>> > > shard
>> > would
>> > > provide a similar speed.
>> > >
>> > > Any way to improve the speed of index rebuild?
>> > >
>> > > Thanks,
>> > > Daniel
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message