lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Drapkin (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-2494) Modify ParallelMultiSearcher to use a CompletionService instead of slowly polling for results
Date Mon, 07 Jun 2010 21:58:11 GMT
Modify ParallelMultiSearcher to use a CompletionService instead of slowly polling for results
---------------------------------------------------------------------------------------------

                 Key: LUCENE-2494
                 URL: https://issues.apache.org/jira/browse/LUCENE-2494
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Search
         Environment: Irrelevant
            Reporter: Edward Drapkin
             Fix For: 3.1
         Attachments: LUCENE-2494.patch

Right now, the parallel multi searcher creates an array/list of Future<V> representing
each of the searchables that's being concurrently searched (and its corresponding search task).

As it stands, once the tasks are all submitted to the executor, the array is iterated over,
FIFO, and Future.get() is called iteratively.  This obviously works, but isn't ideal.  It's
entirely possible (a situation I've run into) where one of the first searchables represents
a large index that takes a long time to search, so the results of the other searchables can't
be processed until the large index is done searching.  In my case, we have two indexes with
several million records that get searched in front of some other indexes, the smallest of
which has only a few ten thousand entries and I didn't think it was ideal for the results
of the other indexes to wait.

I've modified ParallelMultiSearcher to use CompletionServices instead, so that results are
processed in the order they are completed, rather than the order that they are submitted.
 All the tests still pass, and to the best of my knowledge this won't break anything.  This
have several advantages:
1) Speed - the thread owning the executor doesn't have to wait for the first submitted task
to finish in order to process the results of the other tasks, which may have finished first
2) Removed several warnings (even if they are annotated away) due to the ugliness of typecasting
generic arrays.
3) Decreased the complexity of the code in some cases, usually by removing the necessity of
allocating and filling arrays.

With a primed "cache" of searchables, I was getting 700-1200 ms per search, and using the
same phrases, with this patch, I am now getting 400-500ms per search :)

Patch is attached.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message