lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From frer <francois.e...@polymtl.ca>
Subject Re: Query joining 2 indexes
Date Fri, 18 Dec 2009 19:29:10 GMT

Just re-read my post and I don't think it was clear.

The algorithm I ended up doing is:

for all daily data 
        gather hourly ids in a set
        build map with placeholders for hourly values

    get hourly documents from set

    for all daily data
        insert hourly data from documents fetched in placeholders

So the method I showed below is the one to "get hourly documents from set"

François



François Eric wrote:
> 
> Hello,
> 
> I have a performance problem and would need expert advice on how to go 
> about fixing it:
> 
> I currently have 2 indexes: Daily and Hourly.  The Daily index contains 
> about 1,000,000 documents and my Hourly index approximately: 24,000,000 
> documents.  My Daily index contains many fields and some of them are IDs 
> to my Hourly Index.
> What I want to do is fetch data in one request (if possible).
> Right now I do it in many requests:
> 1- Get the matching Daily documents (say it returns 500 documents)
> 2- For each of these documents, locate the Hourly Index Id and fetch it.
> 
> Therefore I make 501 requests to lucene.  This causes some performance 
> issues I guess because of the overhead to  making a request to Lucene.
> 
> Is it possible to do this in 1 request?  I'm thinking no because I'm not 
> sure what the result set would be but maybe I'm missing something.
> 
> If not I guess it would be possible to build a query with my 500 hourly 
> ids and make a OR between them to make it in 2 requests....but then I 
> have to find the matching documents.  Will this overflow if I have 50000 
> ids in my query?
> 
> Anyway, I just want advice on how one would address this situation.
> 
> Thank you very much,
> 
> François
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Query-joining-2-indexes-tp26843980p26848479.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message