Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of kireet@feedly.com designates
 209.85.192.173 as permitted sender)
From: Kireet Reddy <kireet@feedly.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: TieredMergePolicy
Message-Id: <58943385-082A-470D-9BF2-86E256804766@feedly.com>
Date: Sat, 5 Jul 2014 18:47:28 -0700
To: solr-user@lucene.apache.org
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.2\))

I have a question about the maxMergeAtOnce parameter. We are using =
elasticsearch and one of our nodes seems to have very high merge =
activity, However it seems to be high CPU activity and not I/O =
constrainted. I have enabled the IndexWriter info stream logs, and often =
times it seems to do merges of quite small segments (100KB) that are =
much below the floor size (2MB). I suspect this is due to frequent =
refreshes and/or using lots of threads concurrently to do indexing.=20

My supposition is that this is leading to the merge policy doing lots of =
merges of very small segments into another small segment which will =
again require a merge to even reach the floor size. My index has 64 =
segments and 25 are below the floor size. I am wondering if there should =
be an exception for the maxMergesAtOnce parameter for the first level so =
that many small segments could be merged at once in this case?

I am considering changing the other parameters (wider tiers, lower floor =
size, more concurrent merges allowed) but these all seem to have side =
effects I may not necessarily want. Is there a good solution here?=