lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan OConnor <docon...@acquiremedia.com>
Subject is there a way to control when merges happen?
Date Fri, 15 May 2009 20:41:12 GMT
All:

I would like to be able to control when an index merge happens (by wall clock time) so that
merges do not occur in the middle of the business day.

I have a lucene system based on v2.3.2 and we add a couple hundred thousand documents per
day - and we allow searching while documents are being added - we reopen an IndexReader periodically
to expose newly arrived contents.

There are times when merging causes significant performance impacts on search results - I've
seen cases where merging will cause 200% load on a system (dual quad core x86_64 running Centos)
with a raid-5 disk subsystem of 15k drives.

I've seen some info on the MergeScheduler and ConcurrentMergeScheduler but not necessarily
enough to attempt a coding effort.

Looking through the code for ConcurrentMergeScheduler.java, is it as straightforward as over-riding
the mergeScheduler.merge() method with a method that checks to see if a merge is allowed (by
wall clock time)?  If a merge is not allowed at that time, can I just return();? Or do I have
to sleep the thread until the merge is allowed?

Thanks,
Dan


Dan O'Connor
SVP, Engineering
Acquire Media<http://www.acquiremedia.com/>
77 South Bedford Street, Suite 350<http://maps.google.com/maps?f=q&hl=en&geocode=&q=77+S+Bedford+St,+Burlington,+MA+01803&sll=37.0625,-95.677068&sspn=32.472848,80.859375&ie=UTF8&ll=42.485517,-71.197935&spn=0.002287,0.005193&t=h&z=18>
Burlington, MA 01803<http://maps.google.com/maps?f=q&hl=en&geocode=&q=77+S+Bedford+St,+Burlington,+MA+01803&sll=37.0625,-95.677068&sspn=32.472848,80.859375&ie=UTF8&ll=42.485517,-71.197935&spn=0.002287,0.005193&t=h&z=18>
e: doconnor@acquiremedia.com<mailto:doconnor@acquiremedia.com>
o: 781-250-0565
f: 877-861-7724


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message