lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <>
Subject Re: [PMX:FAKE_SENDER] Re: large OR-boolean query
Date Fri, 25 Sep 2009 15:02:12 GMT
This would work a lot better if you did the join at index time. For  
each paper, add a field with all the related drug names (or whatever  
you want to search for), then search on that field.

With the current design, it will never be fast and never scale. Each  
lookup has a cost, so expanding a query to a thousand terms will  
always be slow. Distributing the query to multiple shards will only  
make a bad design slightly faster.

This is fundamental to search index design. The schema is flat, fully- 
denormalized, no joins. You tag each document with the terms that you  
will use to find it. Then you search for those terms directly.


On Sep 25, 2009, at 7:52 AM, Luo, Jeff wrote:

> We are searching strings, not numbers. The reason we are doing this  
> kind
> of query is that we have two big indexes, say, a collection of  
> medicine
> drugs and a collection of research papers. I first run a query against
> the drugs index and get 102400 unique drug names back. Then I need to
> find all the research papers where one or more of the 102400 drug  
> names
> are mentioned, hence the large OR query. This is a kind of JOIN query
> between 2 indexes, which an article in the lucid web site comparing
> databases and search engines briefly touched.
> I was able to issue 100 parallel small queries against solr shards and
> get the results back successfully (even sorted). My custom code is  
> less
> than 100 lines, mostly in my SearchHandler.handleRequestBody. But I  
> have
> problem summing up the correct facet counts because the faceting  
> counts
> from each shard are not disjunctive.
> Based on what is suggested by two other responses to my question, I
> think it is possible that the master can pass the original large query
> to each shard, and each shard will split the large query into 100  
> lower
> level disjunctive lucene queries, fire them against its Lucene index  
> in
> a parallel way and merge the results. Then each shard shall only  
> return
> 1(instead of 100) result set to the master with disjunctive faceting
> counts. It seems that the faceting problem can be solved in this  
> way. I
> would appreciate it if you could let me know if this approach is
> feasible and correct; what solr plug-ins are needed(my guess is a  
> custom
> parser and query-component?)
> Thanks,
> Jeff
> -----Original Message-----
> From: Grant Ingersoll []
> Sent: Thursday, September 24, 2009 10:01 AM
> To:
> Subject: [PMX:FAKE_SENDER] Re: large OR-boolean query
> On Sep 23, 2009, at 4:26 PM, Luo, Jeff wrote:
>> Hi,
>> We are experimenting a parallel approach to issue a large OR-Boolean
>> query, e.g., keywords:(1 OR 2 OR 3 OR ... OR 102400), against several
>> solr shards.
>> The way we are trying is to break the large query into smaller ones,
>> e.g.,
>> the example above can be broken into 10 small queries: keywords:(1
>> OR 2
>> OR 3 OR ... OR 1024), keywords:(1025 OR 1026 OR 1027 OR ... OR 2048),
>> etc
>> Now each shard will get 10 requests and the master will merge the
>> results coming back from each shard, similar to the regular
>> distributed
>> search.
> Can you tell us a little bit more about the why/what of this?  Are you
> really searching numbers or are those just for example?  Do you care
> about the score or do you just need to know whether the result is
> there or not?
> --------------------------
> Grant Ingersoll
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
> using Solr/Lucene:

View raw message