lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <simon.willna...@gmail.com>
Subject Re: Problems with hundreds of BLOCKED threads.
Date Sat, 07 Jul 2012 11:29:45 GMT
On Fri, Jul 6, 2012 at 9:28 PM, Leon Rosenberg <rosenberg.leon@gmail.com> wrote:
> Hello,
>
> we have a small internet shop which uses lucene for product search.
> With increasing traffic we have continuos problem with literaly
> hundreds of threads being BLOCKED in lucene code:
>
> here is an example taken with jstack on prod system:
>
> "ajp-0.0.0.0-8009-1504" daemon prio=10 tid=0x00007f81a518f800
> nid=0x6554 waiting for monitor entry [0x00007f8142c29000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.lucene.store.RAMFile.numBuffers(RAMFile.java:79)
>         - waiting to lock <0x000000054b1daa98> (a org.apache.lucene.store.RAMFile)
>         at org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:87)
>         at org.apache.lucene.store.RAMInputStream.readBytes(RAMInputStream.java:73)
>         at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:82)
>         at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131)
>         at org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:77)
>         at org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:131)
>         at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:139)
>         at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:298)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:524)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:391)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:298)
>
> we have 1172 threads in same state.
>
> there are two threads which already lock the object:
>
> "ajp-0.0.0.0-8009-548" daemon prio=10 tid=0x00007f81a510b000
> nid=0x1c64 waiting for monitor entry [0x00007f81897d5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.lucene.store.RAMFile.getBuffer(RAMFile.java:75)
>         - locked <0x000000054b1daa98> (a org.apache.lucene.store.RAMFile)
>         at org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:97)
>         at org.apache.lucene.store.RAMInputStream.readBytes(RAMInputStream.java:73)
>         at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:82)
>         at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131)
>         at org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:77)
>         at org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:40)
>         at org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:58)
>         at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:296)
>         at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:378)
>         at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:378)
>         at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:589)
>         at org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:167)
>         at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:661)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:298)
>
>
> "ajp-0.0.0.0-8009-663" daemon prio=10 tid=0x00007f81c503d800
> nid=0x225b waiting for monitor entry [0x00007f8182462000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.lucene.store.RAMFile.numBuffers(RAMFile.java:79)
>         - locked <0x000000054b1daa98> (a org.apache.lucene.store.RAMFile)
>         at org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:87)
>         at org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:63)
>         at org.apache.lucene.store.DataInput.readVInt(DataInput.java:105)
>         at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
>         at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131)
>         at org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:77)
>         at org.apache.lucene.search.FilteredTermEnum.setEnum(FilteredTermEnum.java:56)
>         at org.apache.lucene.search.WildcardTermEnum.<init>(WildcardTermEnum.java:65)
>         at org.apache.lucene.search.WildcardQuery.getEnum(WildcardQuery.java:59)
>         at org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:103)
>         at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:139)
>         at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:298)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:524)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:391)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:298)
>
> after some time the BLOCKED threads are waiting to lock another
> object. Over time the amount of locked threads increases.
>
> due to this fact the machine is permanently context-switching and has
> 100% cpu load (vmstat, top).
>
> The responsiveness of search is extremely low, the search needs minutes...
> At the moment of the stacktrace we had approx. 1000 sessions (but the
> problems occured with 400 too) and 1500 threads (we start with 80).
>
> We are reindexing the index every 10 minutes, due to changes in the
> backoffice system.
>
> any ideas what is happening here or what we are doing wrong?
>
> we replaced call to lucene with String.indexOf() to check if the
> problem is in our code, it didn't show the problematic behavior.
>
> Is there a non-blocking search alternative in lucene?

why don't you just use FSDirectory instead of RAMDirectory? Speed is
usually better too and it doesn't have the problems that RAMDirectory
has.

simon
> thanx in advance
> Leon
>
> P.S.
> some data about machine, java, etc
>
>
>
> machine: quadcore, ubuntu linux, 2.6.32, 24Gb RAM
> java version "1.6.0_30"
> Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
>
> memory config:
> jmap -heap 29596
> Attaching to process ID 29596, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.5-b03
>
> using thread-local object allocation.
> Parallel GC with 8 thread(s)
>
> Heap Configuration:
>    MinHeapFreeRatio = 40
>    MaxHeapFreeRatio = 70
>    MaxHeapSize      = 17179869184 (16384.0MB)
>    NewSize          = 1310720 (1.25MB)
>    MaxNewSize       = 17592186044415 MB
>    OldSize          = 5439488 (5.1875MB)
>    NewRatio         = 2
>    SurvivorRatio    = 8
>    PermSize         = 21757952 (20.75MB)
>    MaxPermSize      = 4294967296 (4096.0MB)
>
>
> The machine runs with parallelOldGc and has a full gc every 15-20 minutes.
> The app runs in jboss 6 with jsf.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message