Return-Path: Delivered-To: apmail-jakarta-lucene-user-archive@www.apache.org Received: (qmail 64200 invoked from network); 4 Jun 2004 05:44:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur-2.apache.org with SMTP; 4 Jun 2004 05:44:11 -0000 Received: (qmail 60720 invoked by uid 500); 4 Jun 2004 05:44:26 -0000 Delivered-To: apmail-jakarta-lucene-user-archive@jakarta.apache.org Received: (qmail 60694 invoked by uid 500); 4 Jun 2004 05:44:26 -0000 Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Subscribe: List-Help: List-Post: List-Id: "Lucene Users List" Reply-To: "Lucene Users List" Delivered-To: mailing list lucene-user@jakarta.apache.org Received: (qmail 60677 invoked by uid 99); 4 Jun 2004 05:44:25 -0000 Received: from [203.199.122.36] (HELO web8306.mail.in.yahoo.com) (203.199.122.36) by apache.org (qpsmtpd/0.27.1) with SMTP; Thu, 03 Jun 2004 22:44:25 -0700 Message-ID: <20040604054344.44706.qmail@web8306.mail.in.yahoo.com> Received: from [210.7.75.237] by web8306.mail.in.yahoo.com via HTTP; Fri, 04 Jun 2004 06:43:44 BST Date: Fri, 4 Jun 2004 06:43:44 +0100 (BST) From: =?iso-8859-1?q?Jayant=20Kumar?= Subject: Re: problems with lucene in multithreaded environment To: Lucene Users List In-Reply-To: <40BEA9C7.2000007@apache.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="0-626897822-1086327824=:43546" Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked X-Spam-Rating: minotaur-2.apache.org 1.6.2 0/1000/N --0-626897822-1086327824=:43546 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Content-Id: Content-Disposition: inline We conducted a test on our search for 500 requests given in 27 seconds. We noticed that in the first 5 seconds, the results were coming in 100 to 500 ms. But as the queue size kept increasing, the response time of the search increased drastically to approx 80-100 seconds. Please find enclosed jvmdump.txt which contains a dump of our search program after about 20 seconds of starting the program. Also enclosed is the file queries.txt which contains few sample search queries. Please note that this is done on a sample of 400,000 documents (450MB) on P4 having 1GB RAM. Kindly let us know if this helps to identify the cause of slow response. Jayant --- Doug Cutting wrote: > Jayant Kumar wrote: > > We recently tested lucene with an index size of 2 > GB > > which has about 1,500,000 documents, each document > > having about 25 fields. The frequency of search > was > > about 20 queries per second. This resulted in an > > average response time of about 20 seconds approx > > per search. > > That sounds slow, unless your queries are very > complex. What are your > queries like? > > > What we observed was that lucene queues > > the queries and does not release them until the > > results are found. so the queries that have come > in > > later take up about 500 seconds. Please let us > know > > whether there is a technique to optimize lucene in > > such circumstances. > > Multiple queries executed from different threads > using a single searcher > should not queue, but should run in parallel. A > technique to find out > where threads are queueing is to get a thread dump > and see where all of > the threads are stuck. In Solaris and Linux, > sending the JVM a SIGQUIT > will give a thread dump. On Windows, use > Control-Break. > > Doug > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > lucene-user-unsubscribe@jakarta.apache.org > For additional commands, e-mail: > lucene-user-help@jakarta.apache.org > ________________________________________________________________________ Yahoo! India Matrimony: Find your partner online. http://yahoo.shaadi.com/india-matrimony/ --0-626897822-1086327824=:43546 Content-Type: text/plain; name="jvmdump.txt" Content-Description: jvmdump.txt Content-Disposition: inline; filename="jvmdump.txt" "Thread-14" prio=1 tid=0x080a7420 nid=0x468e waiting for monitor entry [4d61a000..4d61ac18] at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112) - waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51) at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364) at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64) at org.apache.lucene.search.Hits.(Hits.java:43) at org.apache.lucene.search.Searcher.search(Searcher.java:33) at org.apache.lucene.search.Searcher.search(Searcher.java:27) at resdex.searchinc.getHits(searchinc.java:752) at resdex.searchinc.Search(searchinc.java:943) at resdex.searchinctest.conductTestSearch(searchinctest.java:99) at resdex.Server$Handler.run(Server.java:64) at java.lang.Thread.run(Thread.java:534) "Thread-12" prio=1 tid=0x080a58e0 nid=0x468e waiting for monitor entry [4d51a000..4d51ad18] at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112) - waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51) at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364) at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64) at org.apache.lucene.search.Hits.(Hits.java:43) at org.apache.lucene.search.Searcher.search(Searcher.java:33) at org.apache.lucene.search.Searcher.search(Searcher.java:27) at resdex.searchinc.getHits(searchinc.java:752) at resdex.searchinc.Search(searchinc.java:943) at resdex.searchinctest.conductTestSearch(searchinctest.java:99) at resdex.Server$Handler.run(Server.java:64) at java.lang.Thread.run(Thread.java:534) "Thread-11" prio=1 tid=0x080a4b40 nid=0x468e waiting for monitor entry [4d49a000..4d49ad98] at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112) - waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51) at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364) at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64) at org.apache.lucene.search.Hits.(Hits.java:43) at org.apache.lucene.search.Searcher.search(Searcher.java:33) at org.apache.lucene.search.Searcher.search(Searcher.java:27) at resdex.searchinc.getHits(searchinc.java:752) at resdex.searchinc.Search(searchinc.java:943) at resdex.searchinctest.conductTestSearch(searchinctest.java:99) at resdex.Server$Handler.run(Server.java:64) at java.lang.Thread.run(Thread.java:534) "Thread-8" prio=1 tid=0x080a2260 nid=0x468e waiting for monitor entry [4d31a000..4d31af18] at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112) - waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51) at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364) at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64) at org.apache.lucene.search.Hits.(Hits.java:43) at org.apache.lucene.search.Searcher.search(Searcher.java:33) at org.apache.lucene.search.Searcher.search(Searcher.java:27) at resdex.searchinc.getHits(searchinc.java:752) at resdex.searchinc.Search(searchinc.java:943) at resdex.searchinctest.conductTestSearch(searchinctest.java:99) at resdex.Server$Handler.run(Server.java:64) at java.lang.Thread.run(Thread.java:534) "DestroyJavaVM" prio=1 tid=0x08052140 nid=0x468e waiting on condition [0..bfffd004] "Thread-0" prio=1 tid=0x0820cdb0 nid=0x468e runnable [4cf0a000..4cf0a318] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked <0x44ef91e0> (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at resdex.Server.run(Server.java:95) at java.lang.Thread.run(Thread.java:534) "Signal Dispatcher" daemon prio=1 tid=0x0809ce70 nid=0x468e waiting on condition [0..0] "Finalizer" daemon prio=1 tid=0x080884d8 nid=0x468e in Object.wait() [4c921000..4c921598] at java.lang.Object.wait(Native Method) - waiting on <0x44c29be0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) - locked <0x44c29be0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x080878e8 nid=0x468e in Object.wait() [4c8a1000..4c8a1618] at java.lang.Object.wait(Native Method) - waiting on <0x44c29c48> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:429) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115) - locked <0x44c29c48> (a java.lang.ref.Reference$Lock) "VM Thread" prio=1 tid=0x080865c8 nid=0x468e runnable "VM Periodic Task Thread" prio=1 tid=0x0809f680 nid=0x468e waiting on condition "Suspend Checker Thread" prio=1 tid=0x0809c4b8 nid=0x468e runnable --0-626897822-1086327824=:43546 Content-Type: text/plain; name="queries.txt" Content-Description: queries.txt Content-Disposition: inline; filename="queries.txt" Query : ( ( ( ( ( FIELD1: proof OR FIELD2: proof OR FIELD3: proof OR FIELD4: proof OR FIELD5: proof OR FIELD6: proof OR FIELD7: proof ) AND ( FIELD1: "george bush" OR FIELD2: "george bush" OR FIELD3: "george bush" OR FIELD4: "george bush" OR FIELD5: "george bush" OR FIELD6: "george bush" OR FIELD7: "george bush" ) ) AND ( FIELD1: script OR FIELD2: script OR FIELD3: script OR FIELD4: script OR FIELD5: script OR FIELD6: script OR FIELD7: script ) ) AND ( ( FIELD1: san OR FIELD2: san OR FIELD3: san OR FIELD4: san OR FIELD5: san OR FIELD6: san OR FIELD7: san ) OR ( ( FIELD1: war OR FIELD2: war OR FIELD3: war OR FIELD4: war OR FIELD5: war OR FIELD6: war OR FIELD7: war ) OR ( ( FIELD1: gulf OR FIELD2: gulf OR FIELD3: gulf OR FIELD4: gulf OR FIELD5: gulf OR FIELD6: gulf OR FIELD7: gulf ) OR ( ( FIELD1: laden OR FIELD2: laden OR FIELD3: laden OR FIELD4: laden OR FIELD5: laden OR FIELD6: laden OR FIELD7: laden ) OR ( ( FIELD1: ttouristeat OR FIELD2: ttouristeat OR FIELD3: ttouristeat OR FIELD4: ttouristeat OR FIELD5: ttouristeat OR FIELD6: ttouristeat OR FIELD7: ttouristeat ) OR ( ( FIELD1: pow OR FIELD2: pow OR FIELD3: pow OR FIELD4: pow OR FIELD5: pow OR FIELD6: pow OR FIELD7: pow ) OR ( FIELD1: bin OR FIELD2: bin OR FIELD3: bin OR FIELD4: bin OR FIELD5: bin OR FIELD6: bin OR FIELD7: bin ) ) ) ) ) ) ) ) ) AND RANGE: ([ 0800 TO 1100 ]) AND ( S_IDa: (7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR 14 OR 15 OR 16 OR 17 ) or S_IDb: (2 ) ) Time taken : 86304 ms Query : ( ( ( FIELD1: india OR FIELD2: india OR FIELD3: india OR FIELD4: india OR FIELD5: india OR FIELD6: india OR FIELD7: india ) AND ( ( FIELD1: tourist OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR FIELD7: tourist ) OR ( ( FIELD1: attraction OR FIELD2: attraction OR FIELD3: attraction OR FIELD4: attraction OR FIELD5: attraction OR FIELD6: attraction OR FIELD7: attraction ) OR ( ( FIELD1: ps OR FIELD2: ps OR FIELD3: ps OR FIELD4: ps OR FIELD5: ps OR FIELD6: ps OR FIELD7: ps ) OR ( ( FIELD1: pm OR FIELD2: pm OR FIELD3: pm OR FIELD4: pm OR FIELD5: pm OR FIELD6: pm OR FIELD7: pm ) OR ( ( FIELD1: fico OR FIELD2: fico OR FIELD3: fico OR FIELD4: fico OR FIELD5: fico OR FIELD6: fico OR FIELD7: fico ) OR ( ( FIELD1: basis OR FIELD2: basis OR FIELD3: basis OR FIELD4: basis OR FIELD5: basis OR FIELD6: basis OR FIELD7: basis ) OR ( ( FIELD1: abap OR FIELD2: abap OR FIELD3: abap OR FIELD4: abap OR FIELD5: abap OR FIELD6: abap OR FIELD7: abap ) OR ( FIELD1: destination OR FIELD2: destination OR FIELD3: destination OR FIELD4: destination OR FIELD5: destination OR FIELD6: destination OR FIELD7: destination ) ) ) ) ) ) ) ) ) ) AND LOC_A: (27 ) Time taken : 83067 ms Query : ( ( ( ( ( ( ( ( ( FIELD1: usa OR FIELD2: usa OR FIELD3: usa OR FIELD4: usa OR FIELD5: usa OR FIELD6: usa OR FIELD7: usa ) OR ( FIELD1: "vegas / chicago" OR FIELD2: "vegas / chicago" OR FIELD3: "vegas / chicago" OR FIELD4: "vegas / chicago" OR FIELD5: "vegas / chicago" OR FIELD6: "vegas / chicago" OR FIELD7: "vegas / chicago" ) ) AND ( ( FIELD1: " & " OR FIELD2: " & " OR FIELD3: " & " OR FIELD4: " & " OR FIELD5: " & " OR FIELD6: " & " OR FIELD7: " & " ) AND ( ( FIELD1: fun OR FIELD2: fun OR FIELD3: fun OR FIELD4: fun OR FIELD5: fun OR FIELD6: fun OR FIELD7: fun ) AND ( FIELD1: large OR FIELD2: large OR FIELD3: large OR FIELD4: large OR FIELD5: large OR FIELD6: large OR FIELD7: large ) ) ) ) AND ( ( FIELD1: fc OR FIELD2: fc OR FIELD3: fc OR FIELD4: fc OR FIELD5: fc OR FIELD6: fc OR FIELD7: fc ) AND ( ( FIELD1: san OR FIELD2: san OR FIELD3: san OR FIELD4: san OR FIELD5: san OR FIELD6: san OR FIELD7: san ) AND ( ( FIELD1: tourist OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR FIELD7: tourist ) AND ( FIELD1: casino OR FIELD2: casino OR FIELD3: casino OR FIELD4: casino OR FIELD5: casino OR FIELD6: casino OR FIELD7: casino ) ) ) ) ) AND ( ( FIELD1: fc OR FIELD2: fc OR FIELD3: fc OR FIELD4: fc OR FIELD5: fc OR FIELD6: fc OR FIELD7: fc ) OR ( ( FIELD1: san OR FIELD2: san OR FIELD3: san OR FIELD4: san OR FIELD5: san OR FIELD6: san OR FIELD7: san ) OR ( ( FIELD1: tourist OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR FIELD7: tourist ) OR ( FIELD1: casino OR FIELD2: casino OR FIELD3: casino OR FIELD4: casino OR FIELD5: casino OR FIELD6: casino OR FIELD7: casino ) ) ) ) ) AND ( ( FIELD1: fc OR FIELD2: fc OR FIELD3: fc OR FIELD4: fc OR FIELD5: fc OR FIELD6: fc OR FIELD7: fc ) OR ( ( FIELD1: san OR FIELD2: san OR FIELD3: san OR FIELD4: san OR FIELD5: san OR FIELD6: san OR FIELD7: san ) OR ( ( FIELD1: tourist OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR FIELD7: tourist ) OR ( FIELD1: casino OR FIELD2: casino OR FIELD3: casino OR FIELD4: casino OR FIELD5: casino OR FIELD6: casino OR FIELD7: casino ) ) ) ) ) AND ( FIELD1: war OR FIELD2: war OR FIELD3: war OR FIELD4: war OR FIELD5: war OR FIELD6: war OR FIELD7: war ) ) AND ( FIELD1: war OR FIELD2: war OR FIELD3: war OR FIELD4: war OR FIELD5: war OR FIELD6: war OR FIELD7: war ) ) ) AND RANGE: ([ 0000 TO 0100 ]) AND S_IDa: (7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR 14 OR 15 OR 16 OR 17 ) Time taken : 86874 ms Query : ( ( ( ( FIELD1: india OR FIELD2: india OR FIELD3: india OR FIELD4: india OR FIELD5: india OR FIELD6: india OR FIELD7: india ) AND ( ( FIELD1: touristms OR FIELD2: touristms OR FIELD3: touristms OR FIELD4: touristms OR FIELD5: touristms OR FIELD6: touristms OR FIELD7: touristms ) OR ( ( FIELD1: touristis OR FIELD2: touristis OR FIELD3: touristis OR FIELD4: touristis OR FIELD5: touristis OR FIELD6: touristis OR FIELD7: touristis ) OR ( FIELD1: tourist OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR FIELD7: tourist ) ) ) ) AND ( ( FIELD1: h1 OR FIELD2: h1 OR FIELD3: h1 OR FIELD4: h1 OR FIELD5: h1 OR FIELD6: h1 OR FIELD7: h1 ) OR ( ( FIELD1: b1 OR FIELD2: b1 OR FIELD3: b1 OR FIELD4: b1 OR FIELD5: b1 OR FIELD6: b1 OR FIELD7: b1 ) OR ( ( FIELD1: "h1 / b1" OR FIELD2: "h1 / b1" OR FIELD3: "h1 / b1" OR FIELD4: "h1 / b1" OR FIELD5: "h1 / b1" OR FIELD6: "h1 / b1" OR FIELD7: "h1 / b1" ) OR ( ( FIELD1: h OR FIELD2: h OR FIELD3: h OR FIELD4: h OR FIELD5: h OR FIELD6: h OR FIELD7: h ) OR ( FIELD1: "b 1" OR FIELD2: "b 1" OR FIELD3: "b 1" OR FIELD4: "b 1" OR FIELD5: "b 1" OR FIELD6: "b 1" OR FIELD7: "b 1" ) ) ) ) ) ) ) Time taken : 78696 ms Query : ( ( ( FIELD1: relationship OR FIELD2: relationship OR FIELD3: relationship OR FIELD4: relationship OR FIELD5: relationship OR FIELD6: relationship OR FIELD7: relationship ) OR ( ( FIELD1: liability OR FIELD2: liability OR FIELD3: liability OR FIELD4: liability OR FIELD5: liability OR FIELD6: liability OR FIELD7: liability ) OR ( ( FIELD1: banks OR FIELD2: banks OR FIELD3: banks OR FIELD4: banks OR FIELD5: banks OR FIELD6: banks OR FIELD7: banks ) OR ( ( FIELD1: mutual OR FIELD2: mutual OR FIELD3: mutual OR FIELD4: mutual OR FIELD5: mutual OR FIELD6: mutual OR FIELD7: mutual ) OR ( ( FIELD1: funds OR FIELD2: funds OR FIELD3: funds OR FIELD4: funds OR FIELD5: funds OR FIELD6: funds OR FIELD7: funds ) OR ( ( FIELD1: sales OR FIELD2: sales OR FIELD3: sales OR FIELD4: sales OR FIELD5: sales OR FIELD6: sales OR FIELD7: sales ) OR ( FIELD1: hni OR FIELD2: hni OR FIELD3: hni OR FIELD4: hni OR FIELD5: hni OR FIELD6: hni OR FIELD7: hni ) ) ) ) ) ) ) ) AND RANGE: ([ 0200 TO 0900 ]) AND REFIDa: (44 OR 44 OR 45 ) AND C_IDa: (14 ) Time taken : 66183 ms --0-626897822-1086327824=:43546 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org For additional commands, e-mail: lucene-user-help@jakarta.apache.org --0-626897822-1086327824=:43546--