lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shenghua(Daniel) Wan" <wansheng...@gmail.com>
Subject Re: solr/lucene index merge and optimize performance improvement
Date Tue, 16 Jun 2015 16:54:38 GMT
Hi, Toke,
Did you try MapReduce with solr? I think it should be a good fit for your
use case.

On Tue, Jun 16, 2015 at 5:02 AM, Toke Eskildsen <te@statsbiblioteket.dk>
wrote:

> Shenghua(Daniel) Wan <wanshenghua@gmail.com> wrote:
> > Actually, I am currently interested in how to boost merging/optimizing
> > performance of single solr instance.
>
> We have the same challenge (we build static 900GB shards one at a time and
> the final optimization takes 8 hours with only 1 CPU core at 100%). I know
> that there is code for detecting SSDs, which should make merging faster (by
> running more merges in parallel?), but I am afraid that optimize (a single
> merge) is always single threaded.
>
> It seems to me that at least some of the different files making up a
> segment could be created in parallel, but I do not know how hard it would
> be to do so.
>
> - Toke Eskildsen
>



-- 

Regards,
Shenghua (Daniel) Wan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message