incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CGS <cgsmcml...@gmail.com>
Subject Re: Rapidly decreasing insert speed
Date Thu, 15 Mar 2012 15:16:33 GMT
I actually haven't noticed the decrease of the bulk insertion. In my case
it went constantly with 1k per second even for 10M bulk insertions. The
only degradation I noticed when I was forcing bulk insertion to use more of
its buffer (e.g., 800 docs per buffer times 50 buffers dumped at once).
That told me I reached the harddisk capabilities. I think that's why you
got those results at 22k bulk insertion. But, I might be wrong.

CGS




On Thu, Mar 15, 2012 at 2:57 PM, Daniel Gonzalez <gonvaled@gonvaled.com>wrote:

> Thanks,
>
> Something is unclear to me though: if my understanding is correct the
> documents that are bulk created get immediately written to the database (to
> disk). Couchdb actually returns a list specifying which docs have been
> written, and which ones have had problems. In my case, as expected since
> the database is brand-new and no conflicts are expected, all documents get
> correctly written to the database. I take that this means that they have
> been written to disk.
>
> So that means, couchdb buffers are *not* growing: everything is written to
> disk, and an http response is sent as soon as this is done.
>
> Then, if speed is decreasing, we can conclude that couchdb is taking more
> and more time to process the bulk inserts: the increase delay is not due to
> an internally growing buffer in couchdb, but to an internally decreasing
> write-to-disk speed.
>
> Which beggars the question: why is couchdb taking longer and longer to
> write to disk when the database file grows?
>
> Daniel
>
> On Thu, Mar 15, 2012 at 12:39 PM, CGS <cgsmcmlxxv@gmail.com> wrote:
>
> > Hi,
> >
> > It looks fine to me. My tests (on not so great computing element) showed
> a
> > bulk insertion rate peaking at maximum 5-6 kdocs/s, with an average of
> > 1kdocs/s in long run (10 Mdocs). Only that I needed to buffer lower
> number
> > of documents (around 400 buffered documents) and to increase the number
> of
> > buffers (about 50 buffers, but I reduced it after at 25 buffers to align
> my
> > application speed with the average insertion rate) due to the computing
> > element low characteristics (mainly, RAM vs. CPU).
> >
> > In your case, sending more than your harddisk can write down, due to the
> > increasing delay in saving documents, you end up with that degrading rate
> > tail which if you force it further may become ridiculously low. I noticed
> > the same behaviour when I tried to force the bulk insertion rate while I
> > was testing my application capabilities.
> >
> > I hope this piece of information from my experience will help you think
> > your project strategy.
> >
> > Regards,
> > CGS
> >
> >
> >
> >
> > On Thu, Mar 15, 2012 at 12:13 PM, Daniel Gonzalez <gonvaled@gonvaled.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I am processing some input files and inserting the obtained records as
> > > CouchDB documents.
> > > I have noticed that the insert speed is decreasing in pace with
> > > database size increase.
> > >
> > > What I do is:
> > > 1) Read data from input file
> > > 2) Process the data to obtain the structured documents
> > > 3) Put the documents in a local buffer
> > > 4) As soon as the buffer has 1000 documents, perform a couchdb bulk
> > insert
> > > 5) Repeat until input data has been fully processed
> > >
> > > Here you have the log of my current run:
> > >
> > > 2012-03-15 10:15:58,716 - docs= 10000 rate=2282.38 entries/s
> > > 2012-03-15 10:16:46,748 - docs=100000 rate=1822.76 entries/s
> > > 2012-03-15 10:17:47,433 - docs=200000 rate=1592.01 entries/s
> > > 2012-03-15 10:18:48,566 - docs=300000 rate=1358.32 entries/s
> > > 2012-03-15 10:19:54,637 - docs=400000 rate=1572.55 entries/s
> > > 2012-03-15 10:21:01,690 - docs=500000 rate=1560.41 entries/s
> > > 2012-03-15 10:22:09,400 - docs=600000 rate=1556.22 entries/s
> > > 2012-03-15 10:23:16,153 - docs=700000 rate=1550.21 entries/s
> > > 2012-03-15 10:24:30,850 - docs=800000 rate=1393.61 entries/s
> > > 2012-03-15 10:25:46,099 - docs=900000 rate=1336.83 entries/s
> > > 2012-03-15 10:27:09,290 - docs=1000000 rate= 871.37 entries/s
> > > 2012-03-15 10:28:31,745 - docs=1100000 rate=1256.36 entries/s
> > > 2012-03-15 10:29:53,313 - docs=1200000 rate=1140.49 entries/s
> > > 2012-03-15 10:31:29,207 - docs=1300000 rate=1080.79 entries/s
> > > 2012-03-15 10:33:23,917 - docs=1400000 rate= 741.65 entries/s
> > > 2012-03-15 10:35:45,475 - docs=1500000 rate= 567.96 entries/s
> > > 2012-03-15 10:39:04,293 - docs=1600000 rate= 564.01 entries/s
> > > 2012-03-15 10:42:20,160 - docs=1700000 rate= 499.29 entries/s
> > > 2012-03-15 10:46:06,270 - docs=1800000 rate= 505.04 entries/s
> > > 2012-03-15 10:50:24,745 - docs=1900000 rate= 402.14 entries/s
> > > 2012-03-15 10:55:23,800 - docs=2000000 rate= 346.19 entries/s
> > > 2012-03-15 11:02:03,217 - docs=2100000 rate= 274.59 entries/s
> > > 2012-03-15 11:08:21,690 - docs=2200000 rate= 269.57 entries/s
> > >
> > > The "rate" shows the rate of insertion of the last thousand documents,
> > > which as you can see is degrading very fast.
> > >
> > > Regards,
> > > Daniel
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message