lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doron Cohen <DOR...@il.ibm.com>
Subject Re: contrib/benchmark Parallel tasks ?
Date Thu, 18 Oct 2007 08:02:04 GMT
Hi Grant,

Grant Ingersoll wrote:

> I think the answer is:
> [{ "MAddDocs" AddDoc } : 5000] : 4
>
> Is this the functional equivalent of doing:
> { "MAddDocs" AddDoc } : 20000
>
> in parallel?

Yes, this is correct, it reads as "create 4 threads, each
adding 5000 docs to the index, and start/run the 4 threads
in parallel.

Note that a single doc maker is serving all the threads.
So another possible way to run this is:
  [ { AddDoc } : * ] : 4
This reads as: "create 4 threads, each adding docs until
the (single) doc maker is exhausted, and start/run the 4
threads in parallel. This was also discussed in
http://www.mail-archive.com/java-dev@lucene.apache.org/msg10831.html
("threads & benchmark contrib") and so these test cases are also
relevant: TestPerfTasksLogic's testParallelDocMaker()
and testParallelExhausted().

Regards,
Doron

>
> Thanks,
> Grant
>
> On Oct 17, 2007, at 10:42 AM, Grant Ingersoll wrote:
>
> > Hi,
> >
> > I am using the contrib/benchmarker to do some performance tests.  I
> > have a 4 core machine, so I would like to test using 4 threads for
> > indexing.  From the docs at http://lucene.zones.apache.org:8080/
> > hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/benchmark/
> > byTask/package-summary.html
> >
> > I then want to modify the conf/standard.alg file to be just
> > indexing, and to do that in a multithreaded way.
> >
> > I believe this means, I should change:
> > { "MAddDocs" AddDoc } : 20000
> >
> > to be:
> > [ "MAddDocs" AddDoc ] : 20000 : 4
> >
> > right?
> >
> > I guess I am confused a bit by the rate factor documentation in the
> > docs given by the URL above, for instance, it says:
> > "Example - [ AddDoc ] : 400 : 3 - would do 400 addDoc in parallel,
> > starting up to 3 threads per second. "
> >
> > but, I think I want instead: start up 4 threads, and then have each
> > split up the indexing of 20,000 documents.  Thus, each thread would
> > index 5K docs.
> >
> > Am I expressing this correctly?  Something doesn't quite feel right
> > about this.
> >
> > Thanks,
> > Grant
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message