lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elodie Sannier <elodie.sann...@kelkoo.fr>
Subject Re: [Migration Solr5 to Solr6] Unwanted deleted files references
Date Tue, 07 Mar 2017 16:56:54 GMT
Thank you Alex for your answer.

The reference on deleted files are only on index files (with .fdt, .doc.
dvd, ... extensions).

sudo lsof | grep DEL
java       1366    kookel  DEL       REG 253,8               15360013
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_2508z.cfs
java       1366    kookel  DEL       REG 253,8               15360035
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_25091.fdt
java       1366    kookel  DEL       REG 253,8               15425603
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_25091_Lucene50_0.tim
java       1366    kookel  DEL       REG 253,8               11624982
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_2508y.fdt
...

We have tested to optimize the collection with Solr Admin but without
effect on it.

Elodie

On 03/07/2017 04:11 PM, Alexandre Rafalovitch wrote:
> More sanity checks: what are the extensions/types of the files that
> are not deleted?
>
> If they are index files, optimize command (even if no longer
> recommended for production) should really blow all the old ones away.
> So, are they other kinds of files?
>
> Regards,
>     Alex.
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 7 March 2017 at 09:55, Erick Erickson <erickerickson@gmail.com> wrote:
>> Just as a sanity check, if you restart the Solr JVM, do the files
>> disappear from disk?
>>
>> Do you have any custom code anywhere in this chain? If so, do you open
>> any searchers but
>> fail to close them? Although why 6.4 would manifest the problem but
>> other code wouldn't
>> is a mystery, just another sanity check.
>>
>> Best,
>> Erick
>>
>> On Tue, Mar 7, 2017 at 6:44 AM, Elodie Sannier <elodie.sannier@kelkoo.fr> wrote:
>>> Hello,
>>>
>>> We have migrated from Solr 5.4.1 to Solr 6.4.0 and the disk usage has
>>> increased.
>>> We found hundreds of references to deleted index files being held by solr.
>>> Before the migration, we had 15-30% of disk space used, after the migration
>>> we have 60-90% of disk space used.
>>>
>>> We are using Solr Cloud with 2 collections.
>>>
>>> The commands applied on the collections are:
>>> - for incremental indexation mode: add, deleteById with commitWithin of 30
>>> minutes
>>> - for full indexation mode: add, deleteById, commit
>>> - for switch between incremental and full mode: deleteByQuery, createAlias,
>>> reload
>>> - there is also an autocommit every 15 minutes
>>>
>>> We have seen the email "Solr leaking references to deleted files"
>>> 2016-05-31 which describe the same problem but the mentioned bugs are fixed.
>>>
>>> We manually tried to force a commit, a reload and an optimize on the
>>> collections without effect.
>>>
>>> Is a problem of configuration (merge / delete policy) or a possible
>>> regression in the Solr code ?
>>>
>>> Thank you
>>>
>>>
>>> Kelkoo SAS
>>> Société par Actions Simplifiée
>>> Au capital de € 4.168.964,30
>>> Siège social : 158 Ter Rue du Temple 75003 Paris
>>> 425 093 069 RCS Paris
>>>
>>> Ce message et les pièces jointes sont confidentiels et établis à l'attention
>>> exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
>>> message, merci de le détruire et d'en avertir l'expéditeur.


--

Elodie Sannier
Software engineer

<http://www.kelkoo.com/>

*E*elodie.sannier@kelkoo.fr*    Skype*kelkooelodies
*T*+33 (0)4 56 09 07 55
*A*Parc Sud Galaxie, 6, rue des Méridiens, 38130 Echirolles


Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive
de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire
et d'en avertir l'expéditeur.

Mime
View raw message