lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From danielv <dani...@exlibris.co.il>
Subject Merger performance degradation on 3.6.1
Date Wed, 14 May 2014 15:46:11 GMT
Hi,

We have about 550M records index (~800GB) and we merge thousands of mini
indexes once a week using hadoop - 45 mappers on 2 hadoop nodes.
After upgrading to Lucene 3.6.1 we noticed that the merge process
continuously slowing down.
After we test a couple of options it looks like we found the source of the
problem but have no idea how to fix it.
What we do - first we merge all mini-indexes to one intermediate mini-index,
and than this one to the big (final) one.
The difference is deleted_records existence in mini-index:
In case we have no deleted_records from merged mini-indexes - merger run
about 2h with about 05s-2s per mini-index
If we have deleted_records - after about 10 minutes we see dramatic
degradation in time of merging mini-indexes to intermediate one (if first
100-200 mini-indexes merge take less than a second, after 10 minutes is take
more than 10s for one mini-index and after hour or two it is a couple of
minutes!)

This one from jstack of mapper:

   java.lang.Thread.State: RUNNABLE
        at java.lang.Thread.isAlive(Native Method)
        at
org.apache.lucene.util.CloseableThreadLocal.purge(CloseableThreadLocal.java:115)
        - locked <0x00000007db0d6140> (a java.util.WeakHashMap)
        at
org.apache.lucene.util.CloseableThreadLocal.maybePurge(CloseableThreadLocal.java:105)
        at
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:88)
        at
org.apache.lucene.index.TermInfosReader.getThreadResources(TermInfosReader.java:160)
        at
org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:184)
        at
org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:172)
        at
org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:66)
        at
org.apache.lucene.index.BufferedDeletesStream.applyTermDeletes(BufferedDeletesStream.java:346)
        - locked <0x00000007805766f0> (a
org.apache.lucene.index.BufferedDeletesStream)
        at
org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:248)
        - locked <0x00000007805766f0> (a
org.apache.lucene.index.BufferedDeletesStream)
        at
org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3615)
        - locked <0x00000007805739a0> (a
org.apache.lucene.index.IndexWriter)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
        at
org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3120)
        at
org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:3064)

We try to use org.apache.lucene.index.IndexWriter.addIndexes instead of
org.apache.lucene.index.IndexWriter.addIndexesNoOptimize - same behavior.

How can we eliminate this behavior and get improvement in performance of our
merge?

Thanks!



--
View this message in context: http://lucene.472066.n3.nabble.com/Merger-performance-degradation-on-3-6-1-tp4135593.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message