lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-2793) Directory createOutput and openInput should take an IOContext
Date Thu, 02 Jun 2011 15:23:47 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042824#comment-13042824
] 

Michael McCandless commented on LUCENE-2793:
--------------------------------------------

Patch is looking good!  Some feedback:

  * I think CompoundFileReader should not store the IOCtx passed to
    the ctor?  It should just fwd to the dir.openInput...?  It used to
    have to store the readBufferSize in case openInput was called w/o
    a readBufferSize, but we now require IOCtx to openInput.

  * Is IOCtx never allowed to be null?  (I think so?).  In which case
    we should jdoc this, and Dir impls can rely on it.

  * I don't think NRTCachingDir.unCache should take an IOCtx?

  * I think NRTCachineDir.doCacheWrite should take the IOCtx, and, we
    should remove all the merge thread hacks that it now does.

  * The super(bufferSize) call is needed in SimpleFSDir (and others).
    I think what must happen here is SimpleFSDir look @ the IOCtx and
    decides what buffer size to use (the logic in IW should be moved
    down into here), with getters/setters to be able to change
    read/write buffer size for merging vs reading/flushing.

  * IOContext.Context should also have Flush?  And maybe rename Search
    -> Read?

  * IndexWriter ctor shouldn't take an incoming IOCtx; instead,
    IndexWriter should create a new IOCtx each time it opens reader /
    flushes segments / does merging.

  * The ReaderPool.get* methods (inside IndexWriter) should take an
    IOCtx, and the .context from that IOCtx should be part of the key
    used to cache that reader in the pool.


> Directory createOutput and openInput should take an IOContext
> -------------------------------------------------------------
>
>                 Key: LUCENE-2793
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2793
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/store
>            Reporter: Michael McCandless
>            Assignee: Simon Willnauer
>              Labels: gsoc2011, lucene-gsoc-11, mentor
>         Attachments: LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch,
LUCENE-2793.patch
>
>
> Today for merging we pass down a larger readBufferSize than for searching because we
get better performance.
> I think we should generalize this to a class (IOContext), which would hold the buffer
size, but then could hold other flags like DIRECT (bypass OS's buffer cache), SEQUENTIAL,
etc.
> Then, we can make the DirectIOLinuxDirectory fully usable because we would only use DIRECT/SEQUENTIAL
during merging.
> This will require fixing how IW pools readers, so that a reader opened for merging is
not then used for searching, and vice/versa.  Really, it's only all the open file handles
that need to be different -- we could in theory share del docs, norms, etc, if that were somehow
possible.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message