lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <yo...@apache.org>
Subject Re: Realtime Search for Social Networks Collaboration
Date Tue, 09 Sep 2008 14:02:11 GMT
On Tue, Sep 9, 2008 at 5:28 AM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Yonik Seeley wrote:
>> What about something like term freq?  Would it need to count the
>> number of docs after the local maxDoc or is there a better way?
>
> Good question...
>
> I think we'd have to take a full copy of the term -> termFreq on reopen?  I
> don't see how else to do it (I don't understand your suggestion above).  So,
> this will clearly add to the cost of reopen.

One could adjust the freq by iterating over the terms documents...
skipTo(localMaxDoc) and count how many are after that, then subtract
from the freq.  I didn't say it was a *good* idea :-)

>>> For reading stored fields and term vectors, which are now flushed
>>> immediately to disk, we need to somehow get an IndexInput from the
>>> IndexOutputs that IndexWriter holds open on these files.  Or, maybe, just
>>> open new IndexInputs?
>>
>> Hmmm, seems like a case of our nice and simple Directory model not
>> having quite enough features in this case.
>
> I think we can simply open IndexInputs on these files.  I believe Java does
> the right thing on windows, such that if we are already writing to the file,
> it does not prevent another file handle from opening the file for reading.

Yeah, I think the underlying RandomAccessFile might do the right
thing, but IndexInput isn't required to see any changes on the fly
(and current implementations don't) so at a minimum it would be a
change of IndexInput semantics.  Maybe there would need to be a
refresh() function added, or we would need to require a specific
Directory impl?

OR, if all writes are append-only, perhaps we don't ever need to
invalidate the read buffer and would just need to remove the current
logic that caches the file length and then let the underlying
RandomAccessFile do the EOF checking.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message