Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
MIME-Version: 1.0
In-Reply-To: <572494DA.7000208@intix.eu>
References: <CAOcJNz3=FfRQoU4C7TeNRS541J5W58AQ0-V2a9Mz_UN6vOjGRw@mail.gmail.com>
 <CAN4YXvf=ncjag43FmJOOG=R8JB8B9uopZddDNHVRJ5new5Q2XQ@mail.gmail.com>
 <CAOcJNz2CUB1MoY8Vyj9AaxTQ-oaDQi+GftH2ue=KT_EwH1Ed6w@mail.gmail.com> <572494DA.7000208@intix.eu>
From: Erick Erickson <erickerickson@gmail.com>
Date: Mon, 2 May 2016 09:20:39 -0700
Message-ID: <CAN4YXvduf2u1yxeFPeH86He5f=e8eDhAOq=eYnhEJywxpXmt=A@mail.gmail.com>
Subject: Re: Tuning solr for large index with rapid writes
To: solr-user <solr-user@lucene.apache.org>
Content-Type: text/plain; charset=UTF-8
archived-at: Mon, 02 May 2016 16:28:32 -0000

Bram:

That works. I try to monitor the number of 0-hit
queries when I generate a test set on the theory that
those are _usually_ groups of random terms I've
selected that aren't a good model. So it's often
a sequence like "generate my list, see which
ones give 0 results and remove them". Rinse,
repeat.

Like you said, imperfect but _loads_ better than
trying to create them without real user queries
as guidance...

Best,
Erick

On Sat, Apr 30, 2016 at 4:19 AM, Bram Van Dam <bram.vandam@intix.eu> wrote:
>> If I'm reading this right, you have 420M docs on a single shard?
>> Yep, you were reading it right.
>
> Is Erick mentioned, it's hard to give concrete sizing advice, but we've
> found 120M to be the magic number. When a shard contains more than 120M
> documents, performance goes down rapidly & GC pauses grow a lot longer.
> Up until 250M things remain acceptable. But then performance starts to
> drop very quickly after that.
>
>  - Bram
>