lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Le Normand <manuel.lenorm...@gmail.com>
Subject Re: Question about Payloads in Lucene 4.5
Date Mon, 24 Mar 2014 05:06:41 GMT
Hello Rohit,
We had a similar query time bottleneck when attempting to map lucene's
internal id's to the uniqueKey, especially as we generally return only the
uniqueKey to the user we had no other use of the stored field. As you
noted, every internal id --> uniqueKey id requires a disk seek and as in
our use case the documents did not get cached most of these reads were
faulting (random read miss).
There are two patches (for Solr) at SOLR-5478, both make use of docValues
for this need. First one caches the above mapping and a second is more
general and retrieves that mapping (or any other field value) if exists in
fieldCache.
Since implementing the first patch (although not efficiently) we see a
significant performance improvement.

I don't know if these patches can be easily ported to Lucene but I hope
this use case helps you,
Manuel



On Sat, Mar 22, 2014 at 11:45 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Sat, Mar 22, 2014 at 5:18 AM, Rohit Banga <iamrohitbanga@gmail.com>
> wrote:
>
> > Awesome BinaryDocValues sounds nice!
> > I saw that NumericDocValues did not inherit from a base class hence I
> > thought there is no StringDocValues :).
> >
> > Can I expect that a searcher manager will invoke
> > searcherfactory.newSearcher at most once between searcher manager
> > refreshes? I believe IndexSearcher is threadsafe. Is my assumption that
> > newSearcher is invoked only once correct?
>
> It's invoked once, for each refresh.
>
> > If BinaryDocValues didn't exist I was thinking of using a custom searcher
> > factory which would return an instance of a custom subclass of
> > IndexSearcher.This subclass could encapsulate a map from numeric doc
> value
> > to string. I was thinking SearcherManager.acquire could then be used to
> > fetch the instance of this subclass while permitting concurrent updates
> and
> > reads to index and HashMap.
> > Is using SearcherManager in this way appropriate? Just want to make sure
> my
> > understanding of how SearcherManager works is correct.
>
> I think in theory that would be fine (custom IndexSearcher subclass,
> custom map), but hopefully BDV solves this in a simpler way!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message