Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of simon.willnauer@gmail.com
 designates 209.85.219.48 as permitted sender)
MIME-Version: 1.0
Reply-To: simon.willnauer@gmail.com
In-Reply-To: 
 <CAOdYfZU6HLTGQ-qhj7iferpVH7Az0mjFBf-54nvbRdbbbeBPnA@mail.gmail.com>
References: <9007805A-10A6-4F41-8436-B73771375979@googlemail.com>
	<CAOdYfZU6HLTGQ-qhj7iferpVH7Az0mjFBf-54nvbRdbbbeBPnA@mail.gmail.com>
Date: Thu, 8 Nov 2012 11:32:15 +0100
Message-ID: 
 <CAAHmpkhk5CrsL_xUz+_OjOVdyzdR5xSp1k4zUyGofnuehw7eLA@mail.gmail.com>
Subject: Re: Compressed stored fields and multiGet(sorted luceneId[])?
From: Simon Willnauer <simon.willnauer@gmail.com>
To: dev@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Thu, Nov 8, 2012 at 11:30 AM, Robert Muir <rcmuir@gmail.com> wrote:
> Why are you retrieving thousands of stored fields?

really you should roll your own codec for this and specialize. This is
a very unique usecase.

simon
>
> I don't think we should add such an API: stored fields shouldn't be
> used like that, only for things like summary results, so the
> possibility of two documents being in the same block is not so high
> anyway.
>
> And i think the api to stay the way it is (simple visitor) to
> encourage the fact that people shouldnt use stored fields for
> "processing".
>
> On Thu, Nov 8, 2012 at 2:56 AM, eksdev <eksdev@googlemail.com> wrote:
>> Just a theoretical question, would it make sense to add some sort of Sto=
redDocument[] bulkGet(int[] docId) to fetch multiple stored documents in on=
e go?
>>
>> The reasoning behind is that now with compressed blocks random-access ge=
ts more expensive, and in some cases  a user  needs to fetch more documents=
 in one go. If it happens that more documents come from one block it is a w=
in. I would also assume, even without compression , bulk access on sorted d=
ocIds cold be a win (sequential access)?
>>
>> Does that make sense, is it doable? Or even worse, does it already exist=
 :)
>>
>> By the way, I am impressed how well compression does, even on really sho=
rt stored documents, approx. 150b  we observe 35% reduction. Fetching 1000 =
short documents on fully cached index  is observably slower (2-3 times), bu=
t as soon as you memory gets low, compression wins quickly. Did not test it=
 thoroughly, but looks good so far. Great job!
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org