Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: Very high number of deleted docs
From: =?utf-8?Q?Emir_Arnautovi=C4=87?= <emir.arnautovic@sematext.com>
In-Reply-To: <zarafa.59d4d870.6d9d.7751480e6a1ca364@mail1.ams.nl.openindex.io>
Date: Wed, 4 Oct 2017 15:03:06 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <2C4D760D-68E8-4DD7-8713-D7833F2E04CA@sematext.com>
References: <zarafa.59d4d870.6d9d.7751480e6a1ca364@mail1.ams.nl.openindex.io>
To: solr-user@lucene.apache.org
archived-at: Wed, 04 Oct 2017 13:03:20 -0000

Hi Markus,
It is passed but not explicitly - it uses reflection to pass arguments - =
take a look at parent factory class.

When it comes to force merging - you have extreme case - 80% is deleted =
(my guess frequent updates) and extreme cases require some extreme =
measures - it can be either periodic force merge or full reindexing + =
aliases.

HTH,
Emir

--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


> On 4 Oct 2017, at 14:47, Markus Jelsma <markus.jelsma@openindex.io> =
wrote:
>=20
> Do you mean a periodic forceMerge? That is usually considered a bad =
habit on this list (i agree). It is just that i am actually very =
surprised this can happen at all with default settings. This factory, =
unfortunately does not seem to support settings configured in =
solrconfig.
>=20
> Thanks,
> Markus
>=20
> -----Original message-----
>> From:Amrit Sarkar <sarkaramrit2@gmail.com>
>> Sent: Wednesday 4th October 2017 14:42
>> To: solr-user@lucene.apache.org
>> Subject: Re: Very high number of deleted docs
>>=20
>> Hi Markus,
>>=20
>> Emir already mentioned tuning *reclaimDeletesWeight which *affects =
segments
>> about to merge priority. Optimising index time by time, preferably
>> scheduling weekly / fortnight / ..., at low traffic period to never =
be in
>> such odd position of 80% deleted docs in total index.
>>=20
>> Amrit Sarkar
>> Search Engineer
>> Lucidworks, Inc.
>> 415-589-9269
>> www.lucidworks.com
>> Twitter http://twitter.com/lucidworks
>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>>=20
>> On Wed, Oct 4, 2017 at 6:02 PM, Emir Arnautovi=C4=87 <
>> emir.arnautovic@sematext.com> wrote:
>>=20
>>> Hi Markus,
>>> You can set reclaimDeletesWeight in merge settings to some higher =
value
>>> than default (I think it is 2) to favor segments with deleted docs =
when
>>> merging.
>>>=20
>>> HTH,
>>> Emir
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - =
http://sematext.com/
>>>=20
>>>=20
>>>=20
>>>> On 4 Oct 2017, at 13:31, Markus Jelsma <markus.jelsma@openindex.io>
>>> wrote:
>>>>=20
>>>> Hello,
>>>>=20
>>>> Using a 6.6.0, i just spotted one of our collections having a core =
of
>>> which over 80 % of the total number of documents were deleted =
documents.
>>>>=20
>>>> It has <mergePolicyFactory =
class=3D"org.apache.solr.index.TieredMergePolicyFactory"/>
>>> configured with no non-default settings.
>>>>=20
>>>> Is this supposed to happen? How can i prevent these kind of =
numbers?
>>>>=20
>>>> Thanks,
>>>> Markus
>>>=20
>>>=20
>>=20