lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <yo...@apache.org>
Subject Re: Realtime Search for Social Networks Collaboration
Date Tue, 09 Sep 2008 15:51:52 GMT
On Tue, Sep 9, 2008 at 11:42 AM, Ning Li <ning.li.li@gmail.com> wrote:
> On Tue, Sep 9, 2008 at 10:02 AM, Yonik Seeley <yonik@apache.org> wrote:
>> Yeah, I think the underlying RandomAccessFile might do the right
>> thing, but IndexInput isn't required to see any changes on the fly
>> (and current implementations don't) so at a minimum it would be a
>> change of IndexInput semantics.  Maybe there would need to be a
>> refresh() function added, or we would need to require a specific
>> Directory impl?
>>
>> OR, if all writes are append-only, perhaps we don't ever need to
>> invalidate the read buffer and would just need to remove the current
>> logic that caches the file length and then let the underlying
>> RandomAccessFile do the EOF checking.
>
> We cannot assume it's always RandomAccessFile, can we?

No, it would essentially be a change in the semantics that all
implementations would need to support.

> So we may have to flush after writing each document.

Flush when creating a new index view (which could possibly be after
every document is added, but doesn't have to be).

> Even so,
> this may not be sufficient for some FS such as HDFS... Is it
> reasonable in this case to keep in memory everything including
> stored fields and term vectors?

We could maybe do something like a proxy IndexInput/IndexOutput that
would allow updating the read buffer from the writer buffer.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message