lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yoav zibin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-1381) Hanging while indexing/digesting on multiple threads
Date Mon, 02 Jul 2012 18:21:22 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405197#comment-13405197
] 

yoav zibin commented on LUCENE-1381:
------------------------------------

This was fixed in BeanUtil version 1.8.3:
https://issues.apache.org/jira/browse/BEANUTILS-373
                
> Hanging while indexing/digesting on multiple threads
> ----------------------------------------------------
>
>                 Key: LUCENE-1381
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1381
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: modules/analysis
>    Affects Versions: 2.3.2
>         Environment: Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode) on 2.6.9-78.0.1.ELsmp
#1 SMP x86_64 x86_64 x86_64 GNU/Linux
>            Reporter: David Fertig
>
> With several older lucene projects already running and "stable", I have recently written
a multi-threading indexer using to the 2.3.2 release.
> My volume is in the millions of documents indexed daily and I have been stress testing
for a while now.  My current setup has 3 JVMs, each running 6 threads indexing different documents,
with 1 IndexWriter per JVM.  For stability testing, the indexer shutsdown and exits every
5-10 minutes, with a new JVM is started again for a clean restart. At this rate, I have noticed
an rare, but eventually consistent internal hang/deadlock in all indexer threads while parsing
documents.  My 'manager' thread is alive and regularly polling the indexer threads and displaying
their state variables, but the indexer threads themselves appear not to be making progress
while using up nearly 100% of available CPU.  Memory usage is relativly low and stable at
481m out of 2048m available. 
> Most stack traces, and STAY in this state even after repeated inspections: (pressing
CTRL-\ in active JVM window)
> ----------
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode):
> "Thread-6" prio=1 tid=0x0000002b25750920 nid=0x34f6 runnable [0x0000000041465000..0x0000000041465db0]
>         at java.util.WeakHashMap.eq(WeakHashMap.java:254)
>         at java.util.WeakHashMap.get(WeakHashMap.java:345)
>         at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
>         at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
>         at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
>         at org.apache.commons.digester.Rule.end(Rule.java:230)
>         at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
>         at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-5" prio=1 tid=0x0000002b25754eb0 nid=0x34f5 runnable [0x0000000041364000..0x0000000041364d30]
>         at java.lang.String.equals(String.java:858)
>         at org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
>         at java.util.WeakHashMap.eq(WeakHashMap.java:254)
>         at java.util.WeakHashMap.get(WeakHashMap.java:345)
>         at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
>         at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
>         at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
>         at org.apache.commons.digester.Rule.end(Rule.java:230)
>         at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
>         at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-4" prio=1 tid=0x0000002b25754860 nid=0x34f4 runnable [0x0000000041263000..0x0000000041263cb0]
>         at java.lang.String.equals(String.java:858)
>         at org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
>         at java.util.WeakHashMap.eq(WeakHashMap.java:254)
>         at java.util.WeakHashMap.get(WeakHashMap.java:345)
>         at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
>         at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
>         at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
>         at org.apache.commons.digester.Rule.end(Rule.java:230)
>         at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
>         at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-3" prio=1 tid=0x0000002b2509d360 nid=0x34f3 runnable [0x0000000041162000..0x0000000041162c30]
>         at java.lang.String.equals(String.java:858)
>         at org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
>         at java.util.WeakHashMap.eq(WeakHashMap.java:254)
>         at java.util.WeakHashMap.get(WeakHashMap.java:345)
>         at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
>         at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
>         at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
>         at org.apache.commons.digester.Rule.end(Rule.java:230)
>         at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
>         at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-2" prio=1 tid=0x0000002b25083e00 nid=0x34f2 runnable [0x0000000041061000..0x0000000041061bb0]
>         at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289)
>         at java.util.WeakHashMap.getTable(WeakHashMap.java:297)
>         at java.util.WeakHashMap.get(WeakHashMap.java:341)
>         at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
>         at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
>         at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
>         at org.apache.commons.digester.Rule.end(Rule.java:230)
>         at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
>         at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-1" prio=1 tid=0x0000002b2509a4a0 nid=0x34f1 runnable [0x0000000040f60000..0x0000000040f60b30]
>         at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289)
>         at java.util.WeakHashMap.getTable(WeakHashMap.java:297)
>         at java.util.WeakHashMap.get(WeakHashMap.java:341)
>         at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
>         at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
>         at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
>         at org.apache.commons.digester.Rule.end(Rule.java:230)
>         at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
>         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
>         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
>         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
>         at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Low Memory Detector" daemon prio=1 tid=0x0000002b25002430 nid=0x34ef runnable [0x0000000000000000..0x0000000000000000]
> "CompilerThread1" daemon prio=1 tid=0x0000002b250009d0 nid=0x34ee waiting on condition
[0x0000000000000000..0x0000000040c5c6d0]
> "CompilerThread0" daemon prio=1 tid=0x0000002b220c2ce0 nid=0x34ed waiting on condition
[0x0000000000000000..0x0000000040b5b650]
> "AdapterThread" daemon prio=1 tid=0x0000002b220c1750 nid=0x34ec waiting on condition
[0x0000000000000000..0x0000000000000000]
> "Signal Dispatcher" daemon prio=1 tid=0x0000002b220c02d0 nid=0x34eb runnable [0x0000000000000000..0x0000000000000000]
> "Finalizer" daemon prio=1 tid=0x0000002b220ae1e0 nid=0x34ea in Object.wait() [0x0000000040859000..0x0000000040859bb0]
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
>         - locked <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> "Reference Handler" daemon prio=1 tid=0x0000002b220ab980 nid=0x34e9 in Object.wait()
[0x0000000040758000..0x0000000040758b30]
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock)
>         at java.lang.Object.wait(Object.java:474)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>         - locked <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock)
> "main" prio=1 tid=0x0000000040115ea0 nid=0x34e3 waiting on condition [0x0000007fbfffc000..0x0000007fbfffd280]
>         at java.lang.Thread.sleep(Native Method)
> ...
> "VM Thread" prio=1 tid=0x0000002b220a7480 nid=0x34e8 runnable 
> "GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040133fe0 nid=0x34e4 runnable 
> "GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135630 nid=0x34e5 runnable 
> "GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040136480 nid=0x34e6 runnable 
> "GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137300 nid=0x34e7 runnable 
> "VM Periodic Task Thread" prio=1 tid=0x0000002b25004e10 nid=0x34f0 waiting on condition


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message