Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of cgsmcmlxxv@gmail.com designates
 209.85.161.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAOi-OHPsVL57D3n3PzimesM2RpHY1GQ3QRU9pB9HH3FKDtgdQ@mail.gmail.com>
References: 
 <CAAOi-OHPsVL57D3n3PzimesM2RpHY1GQ3QRU9pB9HH3FKDtgdQ@mail.gmail.com>
Date: Thu, 15 Mar 2012 13:39:14 +0200
Message-ID: 
 <CAKrexVJEgnr+AS7TX3Yf8eXnFo-hvuihenmJ0=ODrn51CDJe2Q@mail.gmail.com>
Subject: Re: Rapidly decreasing insert speed
From: CGS <cgsmcmlxxv@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=20cf30363cd19cbbfc04bb468ab9

--20cf30363cd19cbbfc04bb468ab9
Content-Type: text/plain; charset=ISO-8859-1

Hi,

It looks fine to me. My tests (on not so great computing element) showed a
bulk insertion rate peaking at maximum 5-6 kdocs/s, with an average of
1kdocs/s in long run (10 Mdocs). Only that I needed to buffer lower number
of documents (around 400 buffered documents) and to increase the number of
buffers (about 50 buffers, but I reduced it after at 25 buffers to align my
application speed with the average insertion rate) due to the computing
element low characteristics (mainly, RAM vs. CPU).

In your case, sending more than your harddisk can write down, due to the
increasing delay in saving documents, you end up with that degrading rate
tail which if you force it further may become ridiculously low. I noticed
the same behaviour when I tried to force the bulk insertion rate while I
was testing my application capabilities.

I hope this piece of information from my experience will help you think
your project strategy.

Regards,
CGS


On Thu, Mar 15, 2012 at 12:13 PM, Daniel Gonzalez <gonvaled@gonvaled.com>wrote:

> Hi,
>
> I am processing some input files and inserting the obtained records as
> CouchDB documents.
> I have noticed that the insert speed is decreasing in pace with
> database size increase.
>
> What I do is:
> 1) Read data from input file
> 2) Process the data to obtain the structured documents
> 3) Put the documents in a local buffer
> 4) As soon as the buffer has 1000 documents, perform a couchdb bulk insert
> 5) Repeat until input data has been fully processed
>
> Here you have the log of my current run:
>
> 2012-03-15 10:15:58,716 - docs= 10000 rate=2282.38 entries/s
> 2012-03-15 10:16:46,748 - docs=100000 rate=1822.76 entries/s
> 2012-03-15 10:17:47,433 - docs=200000 rate=1592.01 entries/s
> 2012-03-15 10:18:48,566 - docs=300000 rate=1358.32 entries/s
> 2012-03-15 10:19:54,637 - docs=400000 rate=1572.55 entries/s
> 2012-03-15 10:21:01,690 - docs=500000 rate=1560.41 entries/s
> 2012-03-15 10:22:09,400 - docs=600000 rate=1556.22 entries/s
> 2012-03-15 10:23:16,153 - docs=700000 rate=1550.21 entries/s
> 2012-03-15 10:24:30,850 - docs=800000 rate=1393.61 entries/s
> 2012-03-15 10:25:46,099 - docs=900000 rate=1336.83 entries/s
> 2012-03-15 10:27:09,290 - docs=1000000 rate= 871.37 entries/s
> 2012-03-15 10:28:31,745 - docs=1100000 rate=1256.36 entries/s
> 2012-03-15 10:29:53,313 - docs=1200000 rate=1140.49 entries/s
> 2012-03-15 10:31:29,207 - docs=1300000 rate=1080.79 entries/s
> 2012-03-15 10:33:23,917 - docs=1400000 rate= 741.65 entries/s
> 2012-03-15 10:35:45,475 - docs=1500000 rate= 567.96 entries/s
> 2012-03-15 10:39:04,293 - docs=1600000 rate= 564.01 entries/s
> 2012-03-15 10:42:20,160 - docs=1700000 rate= 499.29 entries/s
> 2012-03-15 10:46:06,270 - docs=1800000 rate= 505.04 entries/s
> 2012-03-15 10:50:24,745 - docs=1900000 rate= 402.14 entries/s
> 2012-03-15 10:55:23,800 - docs=2000000 rate= 346.19 entries/s
> 2012-03-15 11:02:03,217 - docs=2100000 rate= 274.59 entries/s
> 2012-03-15 11:08:21,690 - docs=2200000 rate= 269.57 entries/s
>
> The "rate" shows the rate of insertion of the last thousand documents,
> which as you can see is degrading very fast.
>
> Regards,
> Daniel
>

--20cf30363cd19cbbfc04bb468ab9--