Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\))
Subject: Re: optimize status
From: Summer Shire <shiresummer@gmail.com>
In-Reply-To: 
 <CAN4YXvfacT0Wx451Y72p3Siz_80kdu5Vp9S44j0NWBD6KQ8YmA@mail.gmail.com>
Date: Mon, 29 Jun 2015 20:48:27 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <2D7CB74D-EB20-48AD-B7D0-6668FD617CC1@gmail.com>
References: <1ED42941-678F-44ED-AE68-519349172ABE@gmail.com>
 <1435559848.3713228.310202401.7FC93A41@webmail.messagingengine.com>
 <4732223F-99A5-4A27-BEAD-B92DCA691D38@gmail.com>
 <1435565770.1077738.310257457.39CDFE48@webmail.messagingengine.com>
 <CALy4+QVMYRr8HDpz0VSri=Jv3F9XsAt=m9C4bD81Ze_CNZjojA@mail.gmail.com>
 <CAN4YXvfacT0Wx451Y72p3Siz_80kdu5Vp9S44j0NWBD6KQ8YmA@mail.gmail.com>
To: solr-user@lucene.apache.org

Hi Upayavira and Erick,

There are two things we are talking about here.

First: Why am I optimizing? If I don=E2=80=99t our SEARCH (NOT INDEXING) =
performance is 100% worst.=20
The problem lies in the number of total segments. We have to have max =
segments 1 or 2.=20
I have done intensive performance related tests around number of =
segments, merge factor or changing the Merge policy.

Second: Solr does not perform better for me without an optimize. So now =
that I have to optimize the second issue
is updating concurrently during an optimize. If I update when an =
optimize is happening the optimize takes 5 times as long as
the normal optimize.

So is there any way other than creating a postOptimize hook and writing =
the status in a file and somehow making it available to the indexer.=20
All of this just sounds traumatic :)=20

Thanks
Summer


> On Jun 29, 2015, at 5:40 AM, Erick Erickson <erickerickson@gmail.com> =
wrote:
>=20
> Steven:
>=20
> Yes, but....
>=20
> First, here's Mike McCandles' excellent blog on segment merging:
> =
http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.=
html
>=20
> I think the third animation is the TieredMergePolicy. In short, yes an
> optimize will reclaim disk space. But as you update, this is done for
> you anyway. About the only time optimizing is at all beneficial is
> when you have a relatively static index. If you're continually
> updating documents, and by that I mean replacing some existing
> documents, then you'll immediately start generating "holes" in your
> index.
>=20
> And if you _do_ optimize, you wind up with a huge segment. And since
> the default policy tries to merge segments of roughly the same size,
> it accumulates deletes for quite a while before they merged away.
>=20
> And if you don't update existing docs or delete docs, then there's no
> wasted space anyway.
>=20
> Summer:
>=20
> First off, why do you care about not updating during optimizing?
> There's no good reason you have to worry about that, you can freely
> update while optimizing.
>=20
> But frankly I have to agree with Upayavira that on the face of it
> you're doing a lot of extra work. See above, but you optimize while
> indexing, so immediately you're rather defeating the purpose.
> Personally I'd only optimize relatively static indexes and, by
> definition, you're index isn't static since the second process is just
> waiting to modify it.
>=20
> Best,
> Erick
>=20
> On Mon, Jun 29, 2015 at 8:15 AM, Steven White <swhite4141@gmail.com> =
wrote:
>> Hi Upayavira,
>>=20
>> This is news to me that we should not optimize and index.
>>=20
>> What about disk space saving, isn't optimization to reclaim disk =
space or
>> is Solr somehow does that?  Where can I read more about this?
>>=20
>> I'm on Solr 5.1.0 (may switch to 5.2.1)
>>=20
>> Thanks
>>=20
>> Steve
>>=20
>> On Mon, Jun 29, 2015 at 4:16 AM, Upayavira <uv@odoko.co.uk> wrote:
>>=20
>>> I'm afraid I don't understand. You're saying that optimising is =
causing
>>> performance issues?
>>>=20
>>> Simple solution: DO NOT OPTIMIZE!
>>>=20
>>> Optimisation is very badly named. What it does is squashes all =
segments
>>> in your index into one segment, removing all deleted documents. It =
is
>>> good to get rid of deletes - in that sense the index is "optimized".
>>> However, future merges become very expensive. The best way to handle
>>> this topic is to leave it to Lucene/Solr to do it for you. Pretend =
the
>>> "optimize" option never existed.
>>>=20
>>> This is, of course, assuming you are using something like Solr 3.5+.
>>>=20
>>> Upayavira
>>>=20
>>> On Mon, Jun 29, 2015, at 08:08 AM, Summer Shire wrote:
>>>>=20
>>>> Have to cause of performance issues.
>>>> Just want to know if there is a way to tap into the status.
>>>>=20
>>>>> On Jun 28, 2015, at 11:37 PM, Upayavira <uv@odoko.co.uk> wrote:
>>>>>=20
>>>>> Bigger question, why are you optimizing? Since 3.6 or so, it =
generally
>>>>> hasn't been requires, even, is a bad thing.
>>>>>=20
>>>>> Upayavira
>>>>>=20
>>>>>> On Sun, Jun 28, 2015, at 09:37 PM, Summer Shire wrote:
>>>>>> Hi All,
>>>>>>=20
>>>>>> I have two indexers (Independent processes ) writing to a common =
solr
>>>>>> core.
>>>>>> If One indexer process issued an optimize on the core
>>>>>> I want the second indexer to wait adding docs until the optimize =
has
>>>>>> finished.
>>>>>>=20
>>>>>> Are there ways I can do this programmatically?
>>>>>> pinging the core when the optimize is happening is returning OK
>>> because
>>>>>> technically
>>>>>> solr allows you to update when an optimize is happening.
>>>>>>=20
>>>>>> any suggestions ?
>>>>>>=20
>>>>>> thanks,
>>>>>> Summer
>>>=20