Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of greggny3@gmail.com designates
 209.85.219.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAB_8Yd8_pcKLJ4S9PjONxTbx2tpQL97xS_teaEpFNweZP9a8TQ@mail.gmail.com>
References: 
 <CAC=TrKBrxrK-pz4oGUvyyZoamayus8Z7veRkKvy5DO1hPY1KsA@mail.gmail.com>
 <CAB_8Yd8_pcKLJ4S9PjONxTbx2tpQL97xS_teaEpFNweZP9a8TQ@mail.gmail.com>
From: Gregg Donovan <greggny3@gmail.com>
Date: Mon, 3 Mar 2014 11:14:58 -0500
Message-ID: 
 <CAC=TrKDttVEEJNwg423c6Y04=qz3mScd2DTAZ-VvpVyfdMiBaQ@mail.gmail.com>
Subject: Re: Fetching uniqueKey and other int quickly from documentCache?
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=047d7bd6a9aef70b8704f3b61705

--047d7bd6a9aef70b8704f3b61705
Content-Type: text/plain; charset=ISO-8859-1

Yonik,

That's a very clever idea. Unfortunately, I think that will skip the
distributed query optimization we were hoping to take advantage of in
SOLR-1880 [1], but it should work with the proposed distrib.singlePass
optimization in SOLR-5768 [2]. Does that sound right?

--Gregg

[1] https://issues.apache.org/jira/browse/SOLR-1880
[2] https://issues.apache.org/jira/browse/SOLR-5768


On Wed, Feb 26, 2014 at 8:53 PM, Yonik Seeley <yonik@heliosearch.com> wrote:

> You could try forcing things to go through function queries (via
> pseudo-fields):
>
> fl=field(id), field(myfield)
>
> If you're not requesting any stored fields, that *might* currently
> skip that step.
>
> -Yonik
> http://heliosearch.org - native off-heap filters and fieldcache for solr
>
>
> On Mon, Feb 24, 2014 at 9:58 PM, Gregg Donovan <greggny3@gmail.com> wrote:
> > We fetch a large number of documents -- 1000+ -- for each search. Each
> > request fetches only the uniqueKey or the uniqueKey plus one secondary
> > integer key. Despite this, we find that we spent a sizable amount of time
> > in SolrIndexSearcher#doc(int docId, Set<String> fields). Time is spent
> > fetching the two stored fields, LZ4 decoding, etc.
> >
> > I would love to be able to tell Solr to always fetch these two fields
> from
> > memory. We have them both in the fieldCache so we're already spending the
> > RAM. I've seen this asked previously [1], so it seems like a fairly
> common
> > need, especially for distributed search. Any ideas?
> >
> > A few possible ideas I had:
> >
> > --Check FieldCache.html#getCacheEntries() before going to stored fields.
> > --Give the documentCache config a list of fields it should load from the
> > fieldCache
> >
> >
> > Having an in-memory mapping from docId->uniqueKey has come up for us
> > before. We've used a custom SolrCache maintaining that mapping to quickly
> > filter over personalized collections. Maybe the uniqueKey should be more
> > optimized out of the box? Perhaps a custom "uniqueKey" codec that also
> > maintained the docId->uniqueKey mapping in memory?
> >
> > --Gregg
> >
> > [1] http://search-lucene.com/m/oCUKJ1heHUU1
>

--047d7bd6a9aef70b8704f3b61705--