lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen>
Subject Re: Searching on Large Indexes
Date Fri, 27 Jun 2014 11:18:55 GMT
On Fri, 2014-06-27 at 12:33 +0200, Sandeep Khanzode wrote:
> I have an index that runs into 200-300GB. It is not frequently updated.

"not frequently" means different things for different people. Could you
give an approximate time span? If it is updated monthly, you might
consider a full optimization after update.

> What are the best strategies to query on this index?

> 1.] Should I, at index time, split the content, like a hash based
> partition, into multiple separate smaller indexes and aggregate the
> results programmatically?

Assuming you use multiple machines or independent storage for the
multiple indexes, this will bring down latency. Do this if your searches
are too slow.

>  2.] Should I replicate this index and provide some
> sort of document ID, and search on each node for a specific range of
> document IDs?

I don't really follow that idea. Are your searches only ID-based?

Anyway, replication increases throughput. Do this if your server have
trouble keeping up with the full amount of work.

>  3.] Is there any way I can split or move individual
> segments to different nodes and aggregate the results?

Copy the full index. Delete all documents in copy 1 that matches one
half of your ID-hash function, do the reverse for the other. As your
corpus is semi-randomly distributed, scores should be comparable between
the indexes so that the result sets can be easily merged.

But at Jigar says, you should consider switching to SolrCloud (or
ElasticSearch) which does all this for you.

> I am not fully aware of the large scale query strategies. Can you
> please share your findings or experiences?

Depends on what you mean by large scale. You have a running system -
what do you want? Scaling up? Lowering latency? Increasing throughput?
More complex queries?

- Toke Eskildsen, State and University Library, Denmark

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message