lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: Separate issue for appendable field caches / doc values
Date Tue, 23 Aug 2011 19:02:44 GMT
> Oh duh right -- you should be able to alloc too-large arrays and only
> realloc once you run out.  So amortized cost is low but some reopens
> will take a hit to grow....

This hit will be minimal, ie, less than what we're doing now with
cloned deleted docs bit vectors, which feasibly happen on each
getReader() call.  Growing the field cache will occur far less.

Given there isn't a use case for the appendable field cache outside of
RT / LUCENE-2312.  I may bake it in, it's hard to extract, it's hard
to maintain two patches.  However the discussion was good.

> DV is also provided by the IR (perDocValues method) so the RT reader

Ok, it's not clear when / how DVs are used instead of field caches,
and why their access isn't merged together?

On Tue, Aug 23, 2011 at 12:30 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> On Tue, Aug 23, 2011 at 12:09 AM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>>> But, we are trying to move FC under IR control (Martijn has a patch),
>>> at which point an RT IR could have its own appending impl...?
>>
>> LUCENE-3360?  It's placing the field cache into IR / SR.
>
> Yes!
>
>> The RAM
>> reader could return it's own impl where the underlying array can be
>> atomically replaced (when a larger sized array is needed).  I agree
>> that is logical.
>
> Good.
>
>>> But then... FC still returns fixed arrays so you can't append until we fix that?
>>
>> I don't think anything should depend on the size of the field cache
>> array.  If it does, it seems odd.  Are you preferring moving field
>> cache access to method calls?  What is the difference between that and
>> primitive array access?
>
> Oh duh right -- you should be able to alloc too-large arrays and only
> realloc once you run out.  So amortized cost is low but some reopens
> will take a hit to grow....
>
>> For now I will create an independent field cache implementation that
>> is appendable.  It will only be associate-able with the DWPT / RAM
>> reader.  Maybe somehow it can work with LUCENE-3360 and / or the
>> existing static FC access system.
>
> Sounds good.
>
>> Still not sure how doc values fits in.
>
> DV is also provided by the IR (perDocValues method) so the RT reader
> should be able to impl its own.  Each lookup is a method call so it
> should be easier to back that w/ a more RT friendly data structure...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message