lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject DocValues memory usage
Date Tue, 26 Mar 2013 14:59:43 GMT
Inspired by this presentation of DocValues:
http://www.slideshare.net/lucenerevolution/willnauer-simon-doc-values-column-stride-fields-in-lucene
I decided to try them out in 4.2. I created a 1M document index with one
DocValues field:

BinaryDocValuesField conceptsDV = new BinaryDocValuesField("concepts",new
BytesRef(byteArray(4000)));
d.add(conceptsDV);
writer.addDocument(d);

I searched the index and fetched the DocValues field:

TopDocs docs = searcher.search(new TermQuery(new Term("guid", val)), 1);
int docId = docs.scoreDocs[0].doc;
BinaryDocValues conceptValues =
MultiDocValues.getBinaryValues(r,"concepts");
BytesRef result = new BytesRef();
conceptValues.get(docId,result);

However, the first call to MultiDocValues.getBinaryValues reads in the
values for the entire index:

Lucene42DocValuesProducer.loadBinary // loads DocValues for entire index

My hope was to take advantage of faster disk access than stored fields and
less RAM than FieldCache, but this is using too much memory. Are my
assumptions and my usage correct?

Thanks,
Peter

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message