lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan OConnor <docon...@acquiremedia.com>
Subject Re: is there a way to control when merges happen?
Date Fri, 15 May 2009 20:58:16 GMT
Jason,

Thanks for the reply. 

As I was reading the code, it said that if the concurrent merge scheduler ran out of threads,
it ran the merge in the foreground. 

Does that mean the foreground of the merge thread or the indexwriter thread? The former would
be good. The later would seem to be bad. 

Regards,
Dan


----- Original Message -----
From: Jason Rutherglen <jason.rutherglen@gmail.com>
To: java-user@lucene.apache.org <java-user@lucene.apache.org>
Sent: Fri May 15 16:48:54 2009
Subject: Re: is there a way to control when merges happen?

Hi Dan,

You are looking to throttle the merging?  I'd recommend setting
ConcurrentMergeScheduler.setMaxThreadCount(1).  This way IW.addDocument
doesn't wait while a merge occurs (like SerialMergeScheduler) however it
should not use as much CPU as only one merge will occur at a time.

In regards to overriding the MS.merge method either way you mentioned would
work.

-J

On Fri, May 15, 2009 at 1:41 PM, Dan OConnor <doconnor@acquiremedia.com>wrote:

> All:
>
> I would like to be able to control when an index merge happens (by wall
> clock time) so that merges do not occur in the middle of the business day.
>
> I have a lucene system based on v2.3.2 and we add a couple hundred thousand
> documents per day - and we allow searching while documents are being added -
> we reopen an IndexReader periodically to expose newly arrived contents.
>
> There are times when merging causes significant performance impacts on
> search results - I've seen cases where merging will cause 200% load on a
> system (dual quad core x86_64 running Centos) with a raid-5 disk subsystem
> of 15k drives.
>
> I've seen some info on the MergeScheduler and but not necessarily enough to
> attempt a coding effort.
>
> Looking through the code for ConcurrentMergeScheduler.java, is it as
> straightforward as over-riding the mergeScheduler.merge() method with a
> method that checks to see if a merge is allowed (by wall clock time)?  If a
> merge is not allowed at that time, can I just return();? Or do I have to
> sleep the thread until the merge is allowed?
>
> Thanks,
> Dan
>
>
> Dan O'Connor
> SVP, Engineering
> Acquire Media<http://www.acquiremedia.com/>
> 77 South Bedford Street, Suite 350<
> http://maps.google.com/maps?f=q&hl=en&geocode=&q=77+S+Bedford+St,+Burlington,+MA+01803&sll=37.0625,-95.677068&sspn=32.472848,80.859375&ie=UTF8&ll=42.485517,-71.197935&spn=0.002287,0.005193&t=h&z=18
> >
> Burlington, MA 01803<
> http://maps.google.com/maps?f=q&hl=en&geocode=&q=77+S+Bedford+St,+Burlington,+MA+01803&sll=37.0625,-95.677068&sspn=32.472848,80.859375&ie=UTF8&ll=42.485517,-71.197935&spn=0.002287,0.005193&t=h&z=18
> >
> e: doconnor@acquiremedia.com<mailto:doconnor@acquiremedia.com>
> o: 781-250-0565
> f: 877-861-7724
>
>
Mime
View raw message