lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
Date Sun, 15 Jan 2012 19:55:39 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael McCandless updated LUCENE-2795:
---------------------------------------

    Attachment: LUCENE-2795.patch

Hi Varun!

Thanks for checking back here...

I ran this directory on Linux and hit a number of test failures, from
two bugs in the directory impl.  First,
NativeUnixIndexInput.fileLength() failed to include bytes still in the
buffer; second, on clone, if the current II was positioned at EOF
and EOF was on a (512 byte) page boundary then we'd hit
a false IOE.  I fixed those...

I also changed NativeUnixDir to take a delegate Dir which we fall back
to when we don't want to use direct IO; this enabled removing of
open_normal since we just use the "normal" java Dir impls for this.

To run all tests on Linux I do this:
{noformat}
export CLASSPATH=/lucene/unixdir/lucene/build/contrib/misc/lucene-misc-4.0-SNAPSHOT.jar
export LD_LIBRARY_PATH=/lucene/unixdir/lucene/build/native:/usr/local/lib
ant test-core -Dtests.directory=NativeUnixDirectory
{noformat}

(Change the path to your full path... or maybe use relative path but
that could be dangerous since tests may change the CWD... not sure).

It also requires a temporary ctor in the directory taking only File
and using FSDirectory.open for the delegate directory.

Finally I added another setting, minBytesDirect: if the file to open
(in openInput), or expected merge size (in createOutput), is smaller
than this, then we don't use direct IO.

To test this... I temporarily fixed the dir to always use direct IO,
and tests (eventually: ~75 minutes!) passed, except for some Solr tests which look
like timeout problems.  I temporarily dropped the default
buffer size to 4 KB (from 256 KB) else the JVM ran out of direct
buffer space.

I'll test on OS X as well...

                
> Genericize DirectIOLinuxDir -> UnixDir
> --------------------------------------
>
>                 Key: LUCENE-2795
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2795
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/store
>            Reporter: Michael McCandless
>            Assignee: Varun Thacker
>              Labels: gsoc2011, lucene-gsoc-11, mentor
>         Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
LUCENE-2795.patch
>
>
> Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for
indexWriter and not IndexReader (searching).  It's a trap.
> But, once we do LUCENE-2793, we can make it fully general purpose because then a single
native Dir impl can be used.
> I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message