Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 35960 invoked from network); 6 Aug 2009 12:26:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 6 Aug 2009 12:26:23 -0000 Received: (qmail 34060 invoked by uid 500); 6 Aug 2009 12:26:28 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 33999 invoked by uid 500); 6 Aug 2009 12:26:27 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 33989 invoked by uid 99); 6 Aug 2009 12:26:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 12:26:27 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.211.188] (HELO mail-yw0-f188.google.com) (209.85.211.188) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 12:26:19 +0000 Received: by ywh26 with SMTP id 26so1128706ywh.5 for ; Thu, 06 Aug 2009 05:25:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.151.141.16 with SMTP id t16mr17984665ybn.266.1249561556260; Thu, 06 Aug 2009 05:25:56 -0700 (PDT) In-Reply-To: <8163028120305742991D2FB7F19412AB63537A@uksrpblkexb01.detica.com> References: <8163028120305742991D2FB7F19412AB63537A@uksrpblkexb01.detica.com> Date: Thu, 6 Aug 2009 08:25:56 -0400 Message-ID: <9ac0c6aa0908060525k3301c663t8de3204c616f7ab9@mail.gmail.com> Subject: Re: MatchAllDocsQuery concurrency issue From: Michael McCandless To: java-user@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Most likely you're hitting this issue: https://issues.apache.org/jira/browse/LUCENE-1316 Which is fixed in 2.9. Can you try running with 2.9 to confirm? Mike On Thu, Aug 6, 2009 at 8:19 AM, Carl Austin wrote: > Hi, > > I have been seeing an issue running MatchAllDocsQueries concurrently. > Running one against a test index is very fast (70 ms). Running two > concurrently can take 5-25 seconds on the same test index! This issue > doesn't occur with any other type of query I have used. > Because of this, I have put together a few test classes to demonstrate > this in a simple manner. These recreate the issue with only the most > basic Lucene api usage. > > I have incldued the code for the classes below in this email. Just run > TestIndexCreator with a single argument of where to create the test > index. This will create an index with 999999 documents in, each document > having only 1 field with the value "testing" in that field. Only one > unique term for the whole index. > Then run the class Test with the same argument as the index creator. > This will run a 4 tests with a 5 second wait between each. The first > runs a single term query for "testing". The second a single match all > docs query. The third 2 concurrent term queries and the fourth 2 > concurrent match all docs queries. It is the fourth that takes a > relatively long time to complete. > On a larger index I have here it can take 8 minutes to run just 2 match > all docs queries concurrently. > > I look forward to hearing further on this. I can provide this in Jar for > if wished, though I don't know if you can attach to these emails. > > Thanks > > Carl > > > Source Code: > > import org.apache.lucene.analysis.KeywordAnalyzer; > import org.apache.lucene.document.Document; > import org.apache.lucene.document.Field; > import org.apache.lucene.document.Field.Index; > import org.apache.lucene.document.Field.Store; > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.IndexWriter; > > > public class TestIndexCreator { > > > =A0 =A0 =A0 =A0public static void main(String[] args) throws Exception > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (!IndexReader.indexExists(args[0])) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0IndexWriter w =3D new Inde= xWriter(args[0], new > KeywordAnalyzer(), true); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for(int i =3D0; i < 100000= 0; i++) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Document d= oc =3D new Document(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0doc.add(ne= w Field("testfield", > "testing", Store.YES, Index.ANALYZED)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0w.addDocum= ent(doc); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (i % 10= 000 =3D=3D 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0System.out.println(i); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0w.optimize(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0w.close(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > } > > import java.io.IOException; > > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.Term; > import org.apache.lucene.search.IndexSearcher; > import org.apache.lucene.search.MatchAllDocsQuery; > import org.apache.lucene.search.Query; > import org.apache.lucene.search.TermQuery; > import org.apache.lucene.search.TopDocCollector; > > =A0 =A0 =A0 =A0class TestThread implements Runnable > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private IndexReader r; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private int x; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private Test callback; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0private boolean termSearches; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public TestThread(IndexReader r, int x, Te= st callback, > boolean termSearches) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.r =3D r; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.x =3D x; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.callback =3D callback= ; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0this.termSearches =3D term= Searches; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public void run() { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0long time =3D System.curre= ntTimeMillis(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Thread= " + x + " start"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0IndexSearcher s =3D new In= dexSearcher(r); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Query q =3D null; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (termSearches) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0q =3D new = TermQuery(new Term("testfield", > "testing")); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0else > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0q =3D new = MatchAllDocsQuery(); > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0TopDocCollector col =3D ne= w TopDocCollector(50); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0s.search(q= , col); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (IOException e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printSta= ckTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Time t= aken for " + x + " =3D " > + (System.currentTimeMillis() - time)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0callback.callback(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0} > > import java.io.IOException; > > import org.apache.lucene.index.CorruptIndexException; > import org.apache.lucene.index.IndexReader; > > > public class Test { > > =A0 =A0 =A0 =A0private IndexReader r; > =A0 =A0 =A0 =A0private int noSearches =3D 0; > > =A0 =A0 =A0 =A0public static void main(String[] args) > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running Single TermQue= ry Search"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Test t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(1, true); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(5000); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (InterruptedException e1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e1.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running Single MatchAl= lDocs > Search"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(1, false); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(5000); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (InterruptedException e1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e1.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running two concurrent= TermQuery > Searches"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(2, true); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(5000); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (InterruptedException e1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e1.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Running two concurrent= MatchAllDocs > Searches"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t =3D new Test(args[0]); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t.runSearches(2, false); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (Exception e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0public Test(String index) > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0r =3D IndexReader.open(ind= ex); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (CorruptIndexException e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (IOException e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Auto-generated cat= ch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0public void runSearches(int no, boolean termSearches) thro= ws > Exception > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0noSearches =3D no; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for (int i =3D 0; i < no; i++) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thread t1 =3D new Thread(n= ew TestThread(r, i, > this, termSearches)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0t1.start(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0public void callback() > =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0noSearches--; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (noSearches =3D=3D 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0r.close(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (IOException e) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// TODO Au= to-generated catch block > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printSta= ckTrace(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > =A0 =A0 =A0 =A0} > > } > > > > This message should be regarded as confidential. If you have received thi= s email in error please notify the sender and destroy it immediately. > Statements of intent shall only become binding when confirmed in hard cop= y by an authorised signatory. =A0The contents of this email may relate to d= ealings with other companies within the Detica Limited group of companies. > > Detica Limited is registered in England under No: 1337451. > > Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, Eng= land. > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org