lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams" <ch...@manawiz.com>
Subject RE: URGENT: Help indexing large document set
Date Wed, 24 Nov 2004 20:32:52 GMT
Does keyIter return the keys in sorted order?  This should reduce seeks,
especially if the keys are dense.

Also, you should be able to localReader.delete(term) instead of
iterating over the docs (of which I presume there is only one doc since
keys are unique).  This won't improve performance as
IndexReader.delete(Term) does exactly what your code does, but it will
be cleaner.

A linear slowdown with number of docs doesn't make sense, so something
else must be wrong.  I'm not sure what the default buffer size is (it
appears it used to be 128 but is dynamic now I think).  You might find
the slowdown stops after a certain point, especially if you increase
your batch size.

Chuck

  > -----Original Message-----
  > From: John Wang [mailto:john.wang@gmail.com]
  > Sent: Wednesday, November 24, 2004 12:21 PM
  > To: Lucene Users List
  > Subject: Re: URGENT: Help indexing large document set
  > 
  > Thanks Paul!
  > 
  > Using your suggestion, I have changed the update check code to use
  > only the indexReader:
  > 
  > try {
  >               localReader = IndexReader.open(path);
  > 
  >               while (keyIter.hasNext()) {
  >                 key = (String) keyIter.next();
  >                 term = new Term("key", key);
  >                 TermDocs tDocs = localReader.termDocs(term);
  >                 if (tDocs != null) {
  >                   try {
  >                     while (tDocs.next()) {
  >                       localReader.delete(tDocs.doc());
  >                     }
  >                   } finally {
  >                     tDocs.close();
  >                   }
  >                 }
  >               }
  >             } finally {
  > 
  >               if (localReader != null) {
  >                 localReader.close();
  >               }
  > 
  >             }
  > 
  > 
  > Unfortunately it didn't seem to make any dramatic difference.
  > 
  > I also see the CPU is only 30-50% busy, so I am guessing it's
spending
  > a lot of time in IO. Anyway of making the CPU work harder?
  > 
  > Is batch size of 500 too small for 1 million documents?
  > 
  > Currently I am seeing a linear speed degredation of 0.3 milliseconds
  > per document.
  > 
  > Thanks
  > 
  > -John
  > 
  > 
  > On Wed, 24 Nov 2004 09:05:39 +0100, Paul Elschot
  > <paul.elschot@xs4all.nl> wrote:
  > > On Wednesday 24 November 2004 00:37, John Wang wrote:
  > >
  > >
  > > > Hi:
  > > >
  > > >    I am trying to index 1M documents, with batches of 500
documents.
  > > >
  > > >    Each document has an unique text key, which is added as a
  > > > Field.KeyWord(name,value).
  > > >
  > > >    For each batch of 500, I need to make sure I am not adding a
  > > > document with a key that is already in the current index.
  > > >
  > > >   To do this, I am calling IndexSearcher.docFreq for each
document
  > and
  > > > delete the document currently in the index with the same key:
  > > >
  > > >        while (keyIter.hasNext()) {
  > > >             String objectID = (String) keyIter.next();
  > > >             term = new Term("key", objectID);
  > > >             int count = localSearcher.docFreq(term);
  > >
  > > To speed this up a bit make sure that the iterator gives
  > > the terms in sorted order. I'd use an index reader instead
  > > of a searcher, but that will probably not make a difference.
  > >
  > > Adding the documents can be done with multiple threads.
  > > Last time I checked that, there was a moderate speed up
  > > using three threads instead of one on a single CPU machine.
  > > Tuning the values of minMergeDocs and maxMergeDocs
  > > may also help to increase performance of adding documents.
  > >
  > > Regards,
  > > Paul Elschot
  > >
  > >
---------------------------------------------------------------------
  > >
  > >
  > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
  > > For additional commands, e-mail:
lucene-user-help@jakarta.apache.org
  > >
  > >
  > 
  >
---------------------------------------------------------------------
  > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
  > For additional commands, e-mail: lucene-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message