lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: FSDirectory.copy() impl might be dangerous
Date Sat, 26 Jun 2010 05:07:11 GMT
I've Googled around a bit and came across this:
http://markmail.org/message/l67bierbmmedrfw5. Apparently, there's a long
standing bug against SUN since May 2006 (
http://bugs.sun.com/view_bug.do?bug_id=6431344) that's still open and
reports the exact same behavior that I'm seeing.

If I understand correctly, this might be a Windows limitation and is
expected to work well on Linux. I'll give it a try. But this makes me think
if we should keep the current behavior for Linux-based directories, and
fallback to the chunks approach for Windows ones? Since eventually I'll be
running on Linux, I don't want to lose performance ...

This isn't the first that we've witnessed the "write once, run everywhere"
misconception of Java :). I'm thinking if in general we should have a
Windows/Linux FSDirectory impl, or handlers, to prepare for future cases as
well. Mike already started this with LUCENE-2500 (DirectIOLinuxDirectory).
Instead of writing a Directory, perhaps we could have a handler object or
something, or a generic LinuxDirectory that impls some stuff the 'linux'
way. In FSDirectory we already have code which detects the OS and JRE used
to decide between Simple, NIO and MMAP Directories ...

What do you think?

Shai

On Fri, Jun 25, 2010 at 1:48 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> That case was due to this Sun bug:
>
>    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6478546
>
> You'd hit false OOMEs on reading too many bytes at once.
>
> In this case, it looks like transferFrom/To try to mmap the entire
> chunk that you are trying to read, at once!  So if you are copying big
> files this can easily fail on a 32 bit JRE.  So I think we have to
> chunk...
>
> I think 64 MB is too big -- maybe 4 MB (assuming this doesn't cost
> much in perf)?
>
> Mike
>
> On Thu, Jun 24, 2010 at 5:33 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
> > I think we had a similar bug with NIO in NIOFSDir/SimpleFSDir? We have a
> > chunk size there because of that!
> >
> >
> >
> > -----
> >
> > Uwe Schindler
> >
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> >
> > http://www.thetaphi.de
> >
> > eMail: uwe@thetaphi.de
> >
> >
> >
> > From: Shai Erera [mailto:serera@gmail.com]
> > Sent: Thursday, June 24, 2010 11:08 PM
> > To: dev@lucene.apache.org
> > Subject: FSDirectory.copy() impl might be dangerous
> >
> >
> >
> > Hi
> >
> > Today I ran into a weird exception from FSDir.copy(), and while
> > investigating it, I spotted a potential bug as well. So bug first:
> >
> > FileChannel.transferFrom documents that it may not copy the number of
> bytes
> > requested, however we don't check the return value. So need to fix the
> code
> > to read in a loop until all bytes were copied. That's an easy fix.
> >
> > Now for the dangerous part - I wanted to measure segment merging
> > performance, so I created two indexes: 10K docs and 100K docs, both are
> > optimized. I then use IndexWriter.addIndexes(Directory...) method to
> merge
> > 100 copies of the first into a new directory, and 10 copies of the second
> > into a new directory (to create an index of 1M docs, but different number
> of
> > segments). I then call optimize().
> >
> > Surprisingly, when calling addIndexes() w/ the 100K-docs segments, I ran
> > into this exception (Java 1.6 -- Java 1.5's exception was cryptic):
> >
> > Exception in thread "main" java.io.IOException: Map failed
> >     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:770)
> >     at
> >
> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:450)
> >     at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:523)
> >     at org.apache.lucene.store.FSDirectory.copy(FSDirectory.java:450)
> >     at
> org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3019)
> > Caused by: java.lang.OutOfMemoryError: Map failed
> >     at sun.nio.ch.FileChannelImpl.map0(Native Method)
> >     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:767)
> >     ... 7 more
> >
> > I run this on my laptop w/ 4GB RAM. So it's entirely possible there are
> > memory issues here. BUT - the segment size is only 300 MB, which is still
> > much much less than my machine's RAM.
> >
> > What worries me is not that particular test - but what will happen if
> > someone will try to addIndexes() segments that are 10GB or 100GB or even
> > more ... then it really won't matter how much RAM you have. So let's take
> > the RAM availability out of the picture.
> >
> > This API is dangerous, because if someone will try to merge not so large
> > segments, on a machine w/ not so much RAM, he'll hit an exception - and
> it
> > didn't happen before 'cause we used byte[] copies (which is slower).
> >
> > I changed FSDir.copy() code to copy in chunks of 64MB:
> >
> >         long numWritten = 0;
> >         long numToWrite = input.size();
> >         long bufSize = 1 << 26;
> >         while (numWritten < numToWrite) {
> >           numWritten += output.transferFrom(input, numWritten, bufSize);
> >         }
> >
> > And the process completed successfully.
> >
> > Obviously, 64MB may be too high for other systems, so I'm thinking we
> should
> > make it configurable, but still - chunking, using the same API, succeeds.
> I
> > guess it's just a "not so friendly impl" of Java's FileChannelImpl, but I
> > don't know if we can go around it. Maybe we can perf-test and use a
> smaller
> > chunk size that is safer for all cases (and yields the same performance
> as
> > larger ones) ...
> >
> > BTW, I don't have FileChannelImpl's source, but Mike found here
> > http://www.docjar.com/html/api/sun/nio/ch/FileChannelImpl.java.html. It
> > doesn't look like the impl chunks anything ...
> >
> > What do you think?
> >
> > Shai
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Mime
View raw message