lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janne Majaranta <janne.majara...@gmail.com>
Subject SOLR multithreaded faceting
Date Thu, 26 May 2011 12:31:01 GMT
Hi,

I'm experimenting with making field count faceting ("getFacetFieldCounts()")
multithreaded by defining a Callable for each field to run the
faceting-algorithms, and by running these with the ExecutorService having a
fixed threadpool with number of threads =
Runtime.getRuntime().availableProcessors().

In initial tests, when faceting over multiple fields, it seems like a quite
nice speed improvement for counting the facets. And it is a extremely simple
and small addition to the code.

Test index 9M docs / 9 GB, faceting over single valued fields of type
"string" :

A = Solr 3.1
B = Solr 3.1 with parallel per field faceting

In these tests "B" had the index opened via a Windows Network Share from
"A".

A => facet.method=fc, facet field count 8, avg. 840ms
B => facet.method=fc, facet field count 8, avg. 118ms

A => facet field count 8, avg. 120ms
B => facet.method=enum, facet field count 8, avg. 35ms

A => facet.method=fc, facet field count 4, avg. 470ms
B => facet.method=fc, facet field count 4, avg. 80ms

A => facet.method=enum, facet field count 4, avg. 110ms
B => facet.method=enum, facet.field count 4, avg. 30ms

A => facet.method=fc, facet field count 2, avg. 234ms
B => facet.method=fc, facet field count 2, avg. 60ms

A => facet.method=enum, facet field count 2, avg.80ms
B => facet.method=enum, facet field count 2, avg. 40ms

Tested with Chrome repeating CTRL+F5 randomly...

Now, a couple of questions :
- Is somebody already working on adding multithreading to counting the
facets ?
- Is there a reason to not letting the facet counting methods run in
parallel ? i.e. is this a bad idea ?

I see that there is some multithreading work going into SOLR 4.0 with per
segment single valued faceting, is there a bigger plan already on making the
other faceting methods multithreaded also ?

Thanks,

Janne

Mime
View raw message