lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <>
Subject Re: Compressed stored fields and multiGet(sorted luceneId[])?
Date Thu, 08 Nov 2012 10:32:15 GMT
On Thu, Nov 8, 2012 at 11:30 AM, Robert Muir <> wrote:
> Why are you retrieving thousands of stored fields?

really you should roll your own codec for this and specialize. This is
a very unique usecase.

> I don't think we should add such an API: stored fields shouldn't be
> used like that, only for things like summary results, so the
> possibility of two documents being in the same block is not so high
> anyway.
> And i think the api to stay the way it is (simple visitor) to
> encourage the fact that people shouldnt use stored fields for
> "processing".
> On Thu, Nov 8, 2012 at 2:56 AM, eksdev <> wrote:
>> Just a theoretical question, would it make sense to add some sort of StoredDocument[]
bulkGet(int[] docId) to fetch multiple stored documents in one go?
>> The reasoning behind is that now with compressed blocks random-access gets more expensive,
and in some cases  a user  needs to fetch more documents in one go. If it happens that more
documents come from one block it is a win. I would also assume, even without compression ,
bulk access on sorted docIds cold be a win (sequential access)?
>> Does that make sense, is it doable? Or even worse, does it already exist :)
>> By the way, I am impressed how well compression does, even on really short stored
documents, approx. 150b  we observe 35% reduction. Fetching 1000 short documents on fully
cached index  is observably slower (2-3 times), but as soon as you memory gets low, compression
wins quickly. Did not test it thoroughly, but looks good so far. Great job!
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message