lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From david.dav...@correo.aeat.es
Subject Re: About enableLazyFieldLoading and memory
Date Wed, 19 Mar 2014 08:02:04 GMT
That could be an interesting test. Unfortunately now I don't have time to 
do that, but maybe in future.

In order to avoid these memory consumptions we have reduced DocumentCache, 
and we don't have any problems. Besides, big queries that can cause 
problems are never made twice, so the DocumentCache is not needed.

If I have time to check that out I'll post it.

Best regards,

David Dávila Atienza
AEAT - Departamento de Informática Tributaria
Subdirección de Tecnologías de Análisis de la Información e Investigación 
del Fraude
Teléfono: 917681160
Extensión: 30160



De:     Miguel <miguel.valencia@juntadeandalucia.es>
Para:   solr-user@lucene.apache.org, 
Fecha:  19/03/2014 08:35
Asunto: Re: About enableLazyFieldLoading and memory



An interesting check would be disable compression on stored fields, and to 
check if your searcher works better. Disable compression should increase 
stored and searcher should be quicker.

I have read that disable compression all you need to do is to write a new 
codec that uses a stored fields format which does not compress stored 
fields such as Lucene40StoredFieldsFormat.

Best regards

El 18/03/2014 14:47, Shawn Heisey escribió:
On 3/18/2014 7:18 AM, david.davila@correo.aeat.es wrote:

yes, but if I use enableLazyFieldLoading=trueand my queries only request 
for very small fields like ID, DocumentCache shouldn't grow, although my 
stored fields are very big. Am I wrong?


Since Solr 4.1, stored fields are compressed.  This probably means that
in order to get a tiny field out, it must still retrieve an an entire
block of compressed data and uncompress it.

The information in the issue that added the compression feature says
that only one compressed block is ever retrieved for a complete document.

https://issues.apache.org/jira/browse/LUCENE-4226

I wonder if perhaps either Solr or Lucene is dropping all the data into
one or more caches even though you only requested the ID, simply because
it is already available after decompression.  This is only a guess, and
I hope I'm wrong.  If this is indeed happening, it would defeat lazy
field loading.  Can someone with a better understanding comment?

Thanks,
Shawn






Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message