lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randy Tidd>
Subject Document retrieval, performance, and DocValues
Date Tue, 05 Jul 2016 15:20:16 GMT
My Lucene index has about 3 million documents and result sets can be large, often 1000’s
and sometimes as many as 100,000.  I am expecting the index size to grow 5-10x as the system

I index 5 fields, and per recommendations I’ve read, am storing the minimal data in Lucene,
currently just a 12 byte numeric identifier (a Mongo ObjectId) per document.  I store the
rest of the data separately and use the id I get from Lucene to look it up there.

In my load testing, a search like this:

    TopDocs docs =, maxResults, sort)

takes about 50-75 msec which is good.  Retrieving documents with a loop like this:

    for(int i=0; i<docs.scoreDocs.length; i++) {
        ScoreDoc sdoc = docs.scoreDocs[i];
        String id = indexReader.document(sdoc.doc, Collections.singleton("pos_id”)).getField("pos_id").stringValue();
        // … retrieve data with id

takes around 350-400 msec, sometimes as long as 800 msec.  I’m looking for ways to try to
decrease this time if possible.

I’ve read up on DocValues and am not sure if that is intended to help with this.  I understand
that it is a separate store/mapping of Lucene’s internal document id’s to my “pos_id”
which sounds like it may help but I am not sure.  I tried getting the id’s from my reader
like this:

            String id = MultiDocValues.getBinaryValues(indexReader, "pos_id").get(sdoc.doc).utf8ToString()

But performance was no better.  However I saw in the docs for MultiDocValues that I may get
better performance using the "atomic leaves and then operate per-LeafReader”. I searched
around and could not find documentation on how to do that. I see some examples using leaf
readers in the solr projects but they were just examples and I don’t think were written
specifically to optimize performance.  It would be great to find an explanation of why there
are multiple leaf readers per reader and how to use them.

So my questions are 1) are DocValues a possibility for improving my document retrieval performance,
and 2) if so, where can I find an example of this that is written for best performance?

Thanks in advance!


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message