lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-2793) Directory createOutput and openInput should take an IOContext
Date Thu, 09 Jun 2011 15:54:59 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046635#comment-13046635
] 

Michael McCandless commented on LUCENE-2793:
--------------------------------------------

bq. It seems that we don't need to provide IOContext to FieldInfos and SegmentInfo since we
are reading them into memory anyway. I think you can just use a default context here without
changing the constructors. Same is true for SegmentInfo

I think we should pass down "readOnce=true" for these cases?  EG some
kind of caching dir (or something) would know not to bother caching
such files...

Same for del docs, terms index, doc values (well, sometimes), etc.

bq. it seems that we should communicate the IOContext to the codec somehow. I suggest we put
IOContext to SegmentWriteState and SegmentReadState that way we don't need to change the Codec
interface and clutter it with internals. This would also fix mikes comment for FieldsConsumer
etc.

+1 that's great.

bq. I really don't like OneMerge  I think we should add an abstract class (maybe MergeInfo)
that exposes the estimatedMergeBytes, totalDocCount for now.

If we can't include OneMerge, and I agree it'd be nice not to, I think
we should try hard to pull stuff out of OneMerge that may be of
interest to a Dir impl?  Maybe:

  * estimatedTotalSegmentSizeBytes

  * docCount

  * optimize/expungeDeletes

  * isExternal (so Dir can know if this is addIndexes vs "normal" merging)

bq. Regarding the IOContext class I think we should design for what we have right now and
since SegementInfo is not used anywhere (as far as I can see) we should add it once we need
it. OneMerge should not go in there but rather the interface / abstract class I talked about
above.

I agree, let's wait until we have a need.

In fact... SegmentInfo for flush won't work: we go and open all files
for flushing, write to them, close them, and only then do we make the
SegmentInfo.

So it seems like we should also have some abtracted stuff about the
to-be-flushed segment?  Maybe for starters the
estimatedSegmentSizeBytes?  EG, NRTCachingDir could use this to decide
whether to cache the new segment (today it fragile-ly relies on the
app to open new NRT reader frequently enough).


> Directory createOutput and openInput should take an IOContext
> -------------------------------------------------------------
>
>                 Key: LUCENE-2793
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2793
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/store
>            Reporter: Michael McCandless
>            Assignee: Varun Thacker
>              Labels: gsoc2011, lucene-gsoc-11, mentor
>         Attachments: LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch,
LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch
>
>
> Today for merging we pass down a larger readBufferSize than for searching because we
get better performance.
> I think we should generalize this to a class (IOContext), which would hold the buffer
size, but then could hold other flags like DIRECT (bypass OS's buffer cache), SEQUENTIAL,
etc.
> Then, we can make the DirectIOLinuxDirectory fully usable because we would only use DIRECT/SEQUENTIAL
during merging.
> This will require fixing how IW pools readers, so that a reader opened for merging is
not then used for searching, and vice/versa.  Really, it's only all the open file handles
that need to be different -- we could in theory share del docs, norms, etc, if that were somehow
possible.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message