Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 44635 invoked from network); 6 Aug 2009 12:42:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 6 Aug 2009 12:42:39 -0000 Received: (qmail 62584 invoked by uid 500); 6 Aug 2009 12:42:44 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 62513 invoked by uid 500); 6 Aug 2009 12:42:44 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 62503 invoked by uid 99); 6 Aug 2009 12:42:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 12:42:44 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [207.126.144.113] (HELO eu1sys200aog102.obsmtp.com) (207.126.144.113) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 06 Aug 2009 12:42:34 +0000 Received: from source ([193.36.230.160]) by eu1sys200aob102.postini.com ([207.126.147.11]) with SMTP ID DSNKSnrPo0waMcXjxsnXthHuJgvGAZ+SbA/p@postini.com; Thu, 06 Aug 2009 12:42:14 UTC Received: from blackex05.detica.com (unverified) by mail01.detica.com (Content Technologies SMTPRS 4.3.19) with ESMTP id for ; Thu, 6 Aug 2009 13:42:11 +0100 Received: from uksrpblkexb01.detica.com ([10.1.1.38]) by blackex05.detica.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 6 Aug 2009 13:42:10 +0100 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: MatchAllDocsQuery concurrency issue Date: Thu, 6 Aug 2009 13:41:14 +0100 Message-ID: <8163028120305742991D2FB7F19412AB635394@uksrpblkexb01.detica.com> In-Reply-To: <9ac0c6aa0908060525k3301c663t8de3204c616f7ab9@mail.gmail.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: MatchAllDocsQuery concurrency issue Thread-Index: AcoWkSE5FviCmFB4QDOBEK+DYGsDJQAAVGIA From: "Carl Austin" To: X-OriginalArrivalTime: 06 Aug 2009 12:42:10.0298 (UTC) FILETIME=[50ACB5A0:01CA1693] X-Virus-Checked: Checked by ClamAV on apache.org Thanks Mike, Running this with the 2.9 build does resolve the issue it would seem. Unfortunately I can't move to 2.9, especially as it isn't in release = yet. Is there a work-around for 2.4 known that will allow me to get = around this issue as I notice the patches change some underlying classes = such as MultiSegmentReader? What I am doing does regularly have the = chance to call this query concurrently multiple times, which cause quite = an issue. Thanks Carl -----Original Message----- From: Michael McCandless [mailto:lucene@mikemccandless.com]=20 Sent: 06 August 2009 13:26 To: java-user@lucene.apache.org Subject: Re: MatchAllDocsQuery concurrency issue Most likely you're hitting this issue: https://issues.apache.org/jira/browse/LUCENE-1316 Which is fixed in 2.9. Can you try running with 2.9 to confirm? Mike On Thu, Aug 6, 2009 at 8:19 AM, Carl Austin = wrote: > Hi, > > I have been seeing an issue running MatchAllDocsQueries concurrently. > Running one against a test index is very fast (70 ms). Running two=20 > concurrently can take 5-25 seconds on the same test index! This issue=20 > doesn't occur with any other type of query I have used. > Because of this, I have put together a few test classes to demonstrate = > this in a simple manner. These recreate the issue with only the most=20 > basic Lucene api usage. > > I have incldued the code for the classes below in this email. Just run = > TestIndexCreator with a single argument of where to create the test=20 > index. This will create an index with 999999 documents in, each=20 > document having only 1 field with the value "testing" in that field.=20 > Only one unique term for the whole index. > Then run the class Test with the same argument as the index creator. > This will run a 4 tests with a 5 second wait between each. The first=20 > runs a single term query for "testing". The second a single match all=20 > docs query. The third 2 concurrent term queries and the fourth 2=20 > concurrent match all docs queries. It is the fourth that takes a=20 > relatively long time to complete. > On a larger index I have here it can take 8 minutes to run just 2=20 > match all docs queries concurrently. > > I look forward to hearing further on this. I can provide this in Jar=20 > for if wished, though I don't know if you can attach to these emails. > > Thanks > > Carl > > > Source Code: > > import org.apache.lucene.analysis.KeywordAnalyzer; > import org.apache.lucene.document.Document; > import org.apache.lucene.document.Field; import=20 > org.apache.lucene.document.Field.Index; > import org.apache.lucene.document.Field.Store; > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.IndexWriter; > > > public class TestIndexCreator { > > > =A0 =A0 =A0 =A0public static void main(String[] args) throws Exception > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (!IndexReader.indexExists(args[0])) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0IndexWriter w =3D new = IndexWriter(args[0], new=20 > KeywordAnalyzer(), true); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for(int i =3D0; i < = 1000000; i++) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0Document doc =3D new Document(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0doc.add(new Field("testfield",=20 > "testing", Store.YES, Index.ANALYZED)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0w.addDocument(doc); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (i % = 10000 =3D=3D 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0System.out.println(i); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0w.optimize(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0w.close(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > } > > import java.io.IOException; > > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.Term; > import org.apache.lucene.search.IndexSearcher; > import org.apache.lucene.search.MatchAllDocsQuery; > import org.apache.lucene.search.Query; import=20 > org.apache.lucene.search.TermQuery; > import org.apache.lucene.search.TopDocCollector; > > =A0 =A0 =A0 =A0class TestThread implements Runnable > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private IndexReader r; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private int x; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private Test callback; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private boolean termSearches; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public TestThread(IndexReader r, int x, = Test callback,=20 > boolean termSearches) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.r =3D r; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.x =3D x; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.callback =3D = callback; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.termSearches =3D = termSearches; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public void run() { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0long time =3D = System.currentTimeMillis(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0System.out.println("Thread " + x + " start"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0IndexSearcher s =3D new = IndexSearcher(r); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Query q =3D null; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (termSearches) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0q =3D = new TermQuery(new Term("testfield",=20 > "testing")); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0else > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0q =3D = new MatchAllDocsQuery(); > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0TopDocCollector col =3D = new TopDocCollector(50); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0s.search(q, col); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (IOException e) = { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0System.out.println("Time taken for " + x + " =3D " > + (System.currentTimeMillis() - time)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0callback.callback(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0} > > import java.io.IOException; > > import org.apache.lucene.index.CorruptIndexException; > import org.apache.lucene.index.IndexReader; > > > public class Test { > > =A0 =A0 =A0 =A0private IndexReader r; > =A0 =A0 =A0 =A0private int noSearches =3D 0; > > =A0 =A0 =A0 =A0public static void main(String[] args) > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running Single = TermQuery Search"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Test t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(1, true); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(5000); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (InterruptedException e1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e1.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running Single = MatchAllDocs=20 > Search"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(1, = false); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(5000); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (InterruptedException e1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e1.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running two = concurrent TermQuery=20 > Searches"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(2, true); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(5000); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (InterruptedException e1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e1.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running two = concurrent MatchAllDocs=20 > Searches"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(2, = false); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0public Test(String index) > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0r =3D = IndexReader.open(index); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (CorruptIndexException e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (IOException e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated = catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0public void runSearches(int no, boolean termSearches) = throws=20 > Exception > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0noSearches =3D no; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for (int i =3D 0; i < no; i++) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread t1 =3D new = Thread(new TestThread(r, i,=20 > this, termSearches)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t1.start(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0public void callback() > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0noSearches--; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (noSearches =3D=3D 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0r.close(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (IOException e) = { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO = Auto-generated catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > } > > > > This message should be regarded as confidential. If you have received = this email in error please notify the sender and destroy it immediately. > Statements of intent shall only become binding when confirmed in hard = copy by an authorised signatory. =A0The contents of this email may = relate to dealings with other companies within the Detica Limited group = of companies. > > Detica Limited is registered in England under No: 1337451. > > Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, = England. > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org