lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antonio De Miguel <deveto...@gmail.com>
Subject Re: High disk write usage
Date Wed, 05 Jul 2017 16:41:23 GMT
Hi Erik! thanks for your response!

Our soft commit is 5 seconds. Why generates I/0 a softcommit? first notice.


We have enough physical RAM to store full collection and 16Gb for each
JVM.  The collection is relatively small.

I've tried (for testing purposes)  disabling transactionlog (commenting
<updateLog>)... but cluster does not go up. I'll try writing into separated
drive, nice idea...








2017-07-05 18:04 GMT+02:00 Erick Erickson <erickerickson@gmail.com>:

> What is your soft commit interval? That'll cause I/O as well.
>
> How much physical RAM and how much is dedicated to _all_ the JVMs on a
> machine? One cause here is that Lucene uses MMapDirectory which can be
> starved for OS memory if you use too much JVM, my rule of thumb is
> that _at least_ half of the physical memory should be reserved for the
> OS.
>
> Your transaction logs should fluctuate but even out. By that I mean
> they should increase in size but every hard commit should truncate
> some of them so I wouldn't expect them to grow indefinitely.
>
> One strategy is to put your tlogs on a separate drive exactly to
> reduce contention. You could disable them too at a cost of risking
> your data. That might be a quick experiment you could run though,
> disable tlogs and see what that changes. Of course I'd do this on my
> test system ;).
>
> But yeah, Solr will use a lot of I/O in the scenario you are outlining
> I'm afraid.
>
> Best,
> Erick
>
> On Wed, Jul 5, 2017 at 8:08 AM, Antonio De Miguel <devetonez@gmail.com>
> wrote:
> > thanks Markus!
> >
> > We already have SSD.
> >
> > About changing topology.... we probed yesterday with 10 shards, but
> system
> > goes more inconsistent than with the current topology (5x10). I dont know
> > why... too many traffic perhaps?
> >
> > About merge factor.. we set default configuration for some days... but
> when
> > a merge occurs system overload. We probed with mergefactor of 4 to
> improbe
> > query times and trying to have smaller merges.
> >
> > 2017-07-05 16:51 GMT+02:00 Markus Jelsma <markus.jelsma@openindex.io>:
> >
> >> Try mergeFactor of 10 (default) which should be fine in most cases. If
> you
> >> got an extreme case, either create more shards and consider better
> hardware
> >> (SSD's)
> >>
> >> -----Original message-----
> >> > From:Antonio De Miguel <devetonez@gmail.com>
> >> > Sent: Wednesday 5th July 2017 16:48
> >> > To: solr-user@lucene.apache.org
> >> > Subject: Re: High disk write usage
> >> >
> >> > Thnaks a lot alessandro!
> >> >
> >> > Yes, we have very big physical dedicated machines, with a topology of
> 5
> >> > shards and10 replicas each shard.
> >> >
> >> >
> >> > 1. transaction log files are increasing but not with this rate
> >> >
> >> > 2.  we 've probed with values between 300 and 2000 MB... without any
> >> > visible results
> >> >
> >> > 3.  We don't use those features
> >> >
> >> > 4. No.
> >> >
> >> > 5. I've probed with low and high mergefacors and i think that is  the
> >> point.
> >> >
> >> > With low merge factor (over 4) we 've high write disk rate as i said
> >> > previously
> >> >
> >> > with merge factor of 20, writing disk rate is decreasing, but now,
> with
> >> > high qps rates (over 1000 qps) system is overloaded.
> >> >
> >> > i think that's the expected behaviour :(
> >> >
> >> >
> >> >
> >> >
> >> > 2017-07-05 15:49 GMT+02:00 alessandro.benedetti <a.benedetti@sease.io
> >:
> >> >
> >> > > Point 2 was the ram Buffer size :
> >> > >
> >> > > *ramBufferSizeMB* sets the amount of RAM that may be used by Lucene
> >> > >          indexing for buffering added documents and deletions before
> >> they
> >> > > are
> >> > >          flushed to the Directory.
> >> > >          maxBufferedDocs sets a limit on the number of documents
> >> buffered
> >> > >          before flushing.
> >> > >          If both ramBufferSizeMB and maxBufferedDocs is set, then
> >> > >          Lucene will flush based on whichever limit is hit first.
> >> > >
> >> > > <ramBufferSizeMB>100</ramBufferSizeMB>
> >> > > <maxBufferedDocs>1000</maxBufferedDocs>
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > -----
> >> > > ---------------
> >> > > Alessandro Benedetti
> >> > > Search Consultant, R&D Software Engineer, Director
> >> > > Sease Ltd. - www.sease.io
> >> > > --
> >> > > View this message in context: http://lucene.472066.n3.
> >> > > nabble.com/High-disk-write-usage-tp4344356p4344386.html
> >> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >> > >
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message