lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: Token termBuffer issues
Date Tue, 24 Jul 2007 18:40:50 GMT
OK, I ran some benchmarks here.

The performance gains are sizable: 12.8% speedup using Sun's JDK 5 and
17.2% speedup using Sun's JDK 6, on Linux.  This is indexing all
Wikipedia content using LowerCaseTokenizer + StopFilter +
PorterStemFilter.  I think it's worth pursuing!

Here are the optimizations I tested:

  * Change core analyzers to re-use a single Token instance and reuse
    the char[] termBuffer (using a new method "boolean next(Token t)" so it's
    backwards compatible).

  * For the StopFilter I created a new helper class (CharArraySet) to
    create a hash set that can key off of char[]'s without having to
    new a String.

  * Fix the analyzer to re-use the same tokenizer across documents &
    fields (rather than new'ing one every time)

I ran tests with "java -server -Xmx1024M", running on an Intel Core II
Duo box with Debian Linux 2.6.18 kernel and a RAID 5 IO system.

I index all text (every single term) in Wikipedia, pulling from a
single line file (I'm using the patch from LUCENE-947 that adds
line-file creation & indexing to contrib/benchmark).

First I create a single large file that has one doc per line from
Wikipedia content, using this alg:

  docs.dir=enwiki
  doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker
  line.file.out=/lucene/wikifull.txt
  doc.maker.forever=false
  {WriteLineDoc()}: * 

Resulting file is 8.4 GB and 3.2 million docs.  Then I indexed it
using this alg:

  analyzer=org.apache.lucene.analysis.LowercaseStopPorterAnalyzer
  directory=FSDirectory
  ram.flush.mb=64
  max.field.length=2147483647
  compound=false
  max.buffered=70000
  doc.add.log.step=5000
  docs.file=/lucene/wikifull.txt
  doc.maker=org.apache.lucene.benchmark.byTask.feeds.LineDocMaker
  doc.tokenized=true
  doc.maker.forever=false

  ResetSystemErase
  CreateIndex
  { "All"
    {AddDoc}: *
  }

  CloseIndex

  RepSumByPref All
  RepSumByPref AddDoc

Resulting index is 2.2 GB.

The LowercaseStopPorterAnalyzer just looks like this:

  public class LowercaseStopPorterAnalyzer extends Analyzer {
  
    Tokenizer tokenizer;
    TokenStream stream;
    public final TokenStream tokenStream(String fieldName, Reader reader) {
      if (tokenizer == null) {
        tokenizer = new LowerCaseTokenizer(reader);
        stream = new PorterStemFilter(new StopFilter(tokenizer, StopAnalyzer.ENGLISH_STOP_WORDS));
      } else
        tokenizer.reset(reader);
      return stream;
    }
  };

I then record the elapsed time reported by the "All" task.  I ran each
test twice and took the faster time:

  JDK 5 Trunk  21 min 41 sec
  JDK 5   New  18 min 54 sec
    -> 12.8% faster

  JDK 6 Trunk  21 min 43 sec
  JDK 6   New  17 min 59 sec
    -> 17.2% faster

It's rather odd that we see better gains in JDK 6 ... I had expected
the reverse (assuming GC performance is better in JDK 6 than JDK 5).

I also think it's quite cool that we can index all of Wikipedia in 18
minutes :)  That works out to ~8 MB/sec.

I will open an issue...

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message