couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gonzalez <gonva...@gonvaled.com>
Subject Re: Rapidly decreasing insert speed
Date Thu, 15 Mar 2012 12:57:49 GMT
Thanks,

Something is unclear to me though: if my understanding is correct the
documents that are bulk created get immediately written to the database (to
disk). Couchdb actually returns a list specifying which docs have been
written, and which ones have had problems. In my case, as expected since
the database is brand-new and no conflicts are expected, all documents get
correctly written to the database. I take that this means that they have
been written to disk.

So that means, couchdb buffers are *not* growing: everything is written to
disk, and an http response is sent as soon as this is done.

Then, if speed is decreasing, we can conclude that couchdb is taking more
and more time to process the bulk inserts: the increase delay is not due to
an internally growing buffer in couchdb, but to an internally decreasing
write-to-disk speed.

Which beggars the question: why is couchdb taking longer and longer to
write to disk when the database file grows?

Daniel

On Thu, Mar 15, 2012 at 12:39 PM, CGS <cgsmcmlxxv@gmail.com> wrote:

> Hi,
>
> It looks fine to me. My tests (on not so great computing element) showed a
> bulk insertion rate peaking at maximum 5-6 kdocs/s, with an average of
> 1kdocs/s in long run (10 Mdocs). Only that I needed to buffer lower number
> of documents (around 400 buffered documents) and to increase the number of
> buffers (about 50 buffers, but I reduced it after at 25 buffers to align my
> application speed with the average insertion rate) due to the computing
> element low characteristics (mainly, RAM vs. CPU).
>
> In your case, sending more than your harddisk can write down, due to the
> increasing delay in saving documents, you end up with that degrading rate
> tail which if you force it further may become ridiculously low. I noticed
> the same behaviour when I tried to force the bulk insertion rate while I
> was testing my application capabilities.
>
> I hope this piece of information from my experience will help you think
> your project strategy.
>
> Regards,
> CGS
>
>
>
>
> On Thu, Mar 15, 2012 at 12:13 PM, Daniel Gonzalez <gonvaled@gonvaled.com
> >wrote:
>
> > Hi,
> >
> > I am processing some input files and inserting the obtained records as
> > CouchDB documents.
> > I have noticed that the insert speed is decreasing in pace with
> > database size increase.
> >
> > What I do is:
> > 1) Read data from input file
> > 2) Process the data to obtain the structured documents
> > 3) Put the documents in a local buffer
> > 4) As soon as the buffer has 1000 documents, perform a couchdb bulk
> insert
> > 5) Repeat until input data has been fully processed
> >
> > Here you have the log of my current run:
> >
> > 2012-03-15 10:15:58,716 - docs= 10000 rate=2282.38 entries/s
> > 2012-03-15 10:16:46,748 - docs=100000 rate=1822.76 entries/s
> > 2012-03-15 10:17:47,433 - docs=200000 rate=1592.01 entries/s
> > 2012-03-15 10:18:48,566 - docs=300000 rate=1358.32 entries/s
> > 2012-03-15 10:19:54,637 - docs=400000 rate=1572.55 entries/s
> > 2012-03-15 10:21:01,690 - docs=500000 rate=1560.41 entries/s
> > 2012-03-15 10:22:09,400 - docs=600000 rate=1556.22 entries/s
> > 2012-03-15 10:23:16,153 - docs=700000 rate=1550.21 entries/s
> > 2012-03-15 10:24:30,850 - docs=800000 rate=1393.61 entries/s
> > 2012-03-15 10:25:46,099 - docs=900000 rate=1336.83 entries/s
> > 2012-03-15 10:27:09,290 - docs=1000000 rate= 871.37 entries/s
> > 2012-03-15 10:28:31,745 - docs=1100000 rate=1256.36 entries/s
> > 2012-03-15 10:29:53,313 - docs=1200000 rate=1140.49 entries/s
> > 2012-03-15 10:31:29,207 - docs=1300000 rate=1080.79 entries/s
> > 2012-03-15 10:33:23,917 - docs=1400000 rate= 741.65 entries/s
> > 2012-03-15 10:35:45,475 - docs=1500000 rate= 567.96 entries/s
> > 2012-03-15 10:39:04,293 - docs=1600000 rate= 564.01 entries/s
> > 2012-03-15 10:42:20,160 - docs=1700000 rate= 499.29 entries/s
> > 2012-03-15 10:46:06,270 - docs=1800000 rate= 505.04 entries/s
> > 2012-03-15 10:50:24,745 - docs=1900000 rate= 402.14 entries/s
> > 2012-03-15 10:55:23,800 - docs=2000000 rate= 346.19 entries/s
> > 2012-03-15 11:02:03,217 - docs=2100000 rate= 274.59 entries/s
> > 2012-03-15 11:08:21,690 - docs=2200000 rate= 269.57 entries/s
> >
> > The "rate" shows the rate of insertion of the last thousand documents,
> > which as you can see is degrading very fast.
> >
> > Regards,
> > Daniel
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message