lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: Separate issue for appendable field caches / doc values
Date Tue, 23 Aug 2011 20:00:56 GMT
I just examined the field cache code.  I don't think replacing FCs
needs to be difficult.  Lets make the CachedArray values variable
volatile.  values is already public.

On Tue, Aug 23, 2011 at 3:02 PM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
>> Oh duh right -- you should be able to alloc too-large arrays and only
>> realloc once you run out.  So amortized cost is low but some reopens
>> will take a hit to grow....
>
> This hit will be minimal, ie, less than what we're doing now with
> cloned deleted docs bit vectors, which feasibly happen on each
> getReader() call.  Growing the field cache will occur far less.
>
> Given there isn't a use case for the appendable field cache outside of
> RT / LUCENE-2312.  I may bake it in, it's hard to extract, it's hard
> to maintain two patches.  However the discussion was good.
>
>> DV is also provided by the IR (perDocValues method) so the RT reader
>
> Ok, it's not clear when / how DVs are used instead of field caches,
> and why their access isn't merged together?
>
> On Tue, Aug 23, 2011 at 12:30 PM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> On Tue, Aug 23, 2011 at 12:09 AM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>>>> But, we are trying to move FC under IR control (Martijn has a patch),
>>>> at which point an RT IR could have its own appending impl...?
>>>
>>> LUCENE-3360?  It's placing the field cache into IR / SR.
>>
>> Yes!
>>
>>> The RAM
>>> reader could return it's own impl where the underlying array can be
>>> atomically replaced (when a larger sized array is needed).  I agree
>>> that is logical.
>>
>> Good.
>>
>>>> But then... FC still returns fixed arrays so you can't append until we fix
that?
>>>
>>> I don't think anything should depend on the size of the field cache
>>> array.  If it does, it seems odd.  Are you preferring moving field
>>> cache access to method calls?  What is the difference between that and
>>> primitive array access?
>>
>> Oh duh right -- you should be able to alloc too-large arrays and only
>> realloc once you run out.  So amortized cost is low but some reopens
>> will take a hit to grow....
>>
>>> For now I will create an independent field cache implementation that
>>> is appendable.  It will only be associate-able with the DWPT / RAM
>>> reader.  Maybe somehow it can work with LUCENE-3360 and / or the
>>> existing static FC access system.
>>
>> Sounds good.
>>
>>> Still not sure how doc values fits in.
>>
>> DV is also provided by the IR (perDocValues method) so the RT reader
>> should be able to impl its own.  Each lookup is a method call so it
>> should be easier to back that w/ a more RT friendly data structure...
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message