lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doron Cohen <>
Subject Re: threads & benchmark contrib
Date Sat, 23 Jun 2007 06:22:34 GMT
Mike, I didn't anticipate this use case and I think it
would not work correctly. I'll look into this.

Anyhow, I think it would not work as you expect.

It seems what you want is to have 4 threads, adding docs in
parallel, until the doc maker is exhausted.

But this line:
  {[AddDoc(4000)]: 4} : *

Reads as -
  Repeatedly until exhausted:
     Create & Start 4 threads (in parallel),
       each adding 1 doc of size 4000;
     Wait for them 4 threads to complete.

Now, this is not what you are after, is it? I think
you would like just 4 threads to do all the work.

It seems what you are really after is this:
   [ { AddDoc } : * ] : 4

This reads as:
  Create 4 threads, each adding docs until exhaustion.

Since there is a single system-benchmark-wide doc-maker, all 4
threads use it, and when it is exhausted, all 4 will be done.

I tried this way and it works as I expected it to (except
for that DateFormat bug, see below). Can you try like this
and let me know if it works for you.

I think your variation of this exposes a bug in the
benchmark - it will just loop forever because the parallel
sequence would mask the exhaustion from the outer sequential
sequence. I opened LUCENE-941 for this, and looking into it


"Michael McCandless" <> wrote on 22/06/2007
> Hi,
> I'm trying to test LUCENE-843 (IndexWriter speedups) on Wikipedia
> using the the benchmark contrib framework plus the patch from
> LUCENE-848.
> I downloaded an older wikipedia export (the "latest" doesn't seem to
> exist) and got it un-tar'd.  The test I'd like to run is to use 4
> threads to index all (exhaust) documents.  I'm using the alg below.
> One problem I hit is the DirDocMaker uses a SimpleDateFormat instance
> for parsing the dates at the top of each file, but, this is not
> threadsafe and so I hit exceptions from there.  I think we just need
> to make that instance thread local I think (I will open issue).

Yes, tha's a bug...  It is also in some already committed parts
of the benchmark. I opened LUCENE-940 for this.

> The question I have is: is this alg going to do what I want?  I'd like
> each doc in Wikipedia to be indexed only once, with 4 threads running.
> I *think* but I'm not sure that the alg below actually indexes the
> Wikipedia content 4 times over instead?
> Here's the alg:
> max.field.length=2147483647
> compound=false
> analyzer=org.apache.lucene.analysis.SimpleAnalyzer
> directory=FSDirectory
> # ram.flush.mb=32
> max.buffered=10000
> doc.stored=true
> doc.tokenized=true
> doc.term.vector=true
> doc.add.log.step=500
> docs.dir=enwiki
> doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker
> # task at this depth or less would print when they start
> task.max.depth.log=1
> doc.maker.forever=false
> #

> ResetSystemErase
> CreateIndex
> {[AddDoc(4000)]: 4} : *
> CloseIndex
> RepSumByPref AddDoc
> Mike
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message