lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shalin Shekhar Mangar <shalinman...@gmail.com>
Subject Re: Suggester uses lots of 'Page cache' memory
Date Tue, 02 May 2017 08:16:54 GMT
On Tue, May 2, 2017 at 10:22 AM, Damien Kamerman <damienk@gmail.com> wrote:
> Hi all,
>
> I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each replica
> uses about 14GB disk usage. I'm using Solaris 11 and I see the 'Page cache'
> grow by about 7GB for each suggester replica I build. The suggester index
> itself is very small. The 'Page cache' memory is freed when the node is
> stopped.
>
> I guess the Suggester component is mmap'ing the entire Lucene index into
> memory and holding it? Is this expected behavior? Is there a workaround?
>

Yes, this is expected. The suggester opens the index using Lucene's
MMapDirectory which ends up memory mapping the index. But the memory
mapped pages can be shared across everyone using the same index which
basically means that the replica's usual index searcher can also use
these pages. If you were not building the suggester index, even then
the replica's search index would have been mmapped but perhaps only on
demand instead of all at once.

> I use this command to build the suggester for just the replica
> 'target1_shard1_replica1':
> curl "
> http://localhost:8983/solr/collection1/suggest?suggest.dictionary=mySuggester&suggest.build=true&shards=localhost:8983/solr/target1_shard1_replica1
> "
>
> BTW: Without the 'shards' param the distributed request will randomly hit
> half the replicas.

Yes, this is a problem. I recently opened
https://issues.apache.org/jira/browse/SOLR-10532 but unfortunately I
don't have the time to fix it soon. Patches are always welcome.

>
> From my solrconfig.xml:
> <searchComponent name="suggest" class="solr.SuggestComponent">
> <lst name="suggester">
> <str name="name">mySuggester</str>
> <str name="lookupImpl">AnalyzingInfixLookupFactory</str>
> <str name="indexPath">mySuggester</str>
> <str name="dictionaryImpl">DocumentDictionaryFactory</str>
> <str name="field">mySuggest</str>
> <str name="contextField">x</str>
> <str name="suggestAnalyzerFieldType">suggestTypeLc</str>
> <str name="buildOnStartup">false</str>
> </lst>
> </searchComponent>
>
> Cheers,
> Damien.



-- 
Regards,
Shalin Shekhar Mangar.

Mime
View raw message