Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <000901cb13e4$e2809560$a781c020$@thetaphi.de>
References: <AANLkTimzgDY-rr85DO7uzb-wPaQaEuPhKQzrtVOUAPmW@mail.gmail.com>
	<000901cb13e4$e2809560$a781c020$@thetaphi.de>
Date: Thu, 24 Jun 2010 18:48:39 -0400
Message-ID: <AANLkTimJnQyHH2c4nMKsCQFD51ONzW3Xi0t2tsI-GiI7@mail.gmail.com>
Subject: Re: FSDirectory.copy() impl might be dangerous
From: Michael McCandless <lucene@mikemccandless.com>
To: dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

That case was due to this Sun bug:

    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=3D6478546

You'd hit false OOMEs on reading too many bytes at once.

In this case, it looks like transferFrom/To try to mmap the entire
chunk that you are trying to read, at once!  So if you are copying big
files this can easily fail on a 32 bit JRE.  So I think we have to
chunk...

I think 64 MB is too big -- maybe 4 MB (assuming this doesn't cost
much in perf)?

Mike

On Thu, Jun 24, 2010 at 5:33 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
> I think we had a similar bug with NIO in NIOFSDir/SimpleFSDir? We have a
> chunk size there because of that!
>
>
>
> -----
>
> Uwe Schindler
>
> H.-H.-Meier-Allee 63, D-28213 Bremen
>
> http://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> From: Shai Erera [mailto:serera@gmail.com]
> Sent: Thursday, June 24, 2010 11:08 PM
> To: dev@lucene.apache.org
> Subject: FSDirectory.copy() impl might be dangerous
>
>
>
> Hi
>
> Today I ran into a weird exception from FSDir.copy(), and while
> investigating it, I spotted a potential bug as well. So bug first:
>
> FileChannel.transferFrom documents that it may not copy the number of byt=
es
> requested, however we don't check the return value. So need to fix the co=
de
> to read in a loop until all bytes were copied. That's an easy fix.
>
> Now for the dangerous part - I wanted to measure segment merging
> performance, so I created two indexes: 10K docs and 100K docs, both are
> optimized. I then use IndexWriter.addIndexes(Directory...) method to merg=
e
> 100 copies of the first into a new directory, and 10 copies of the second
> into a new directory (to create an index of 1M docs, but different number=
 of
> segments). I then call optimize().
>
> Surprisingly, when calling addIndexes() w/ the 100K-docs segments, I ran
> into this exception (Java 1.6 -- Java 1.5's exception was cryptic):
>
> Exception in thread "main" java.io.IOException: Map failed
> =A0=A0=A0 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:770)
> =A0=A0=A0 at
> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:=
450)
> =A0=A0=A0 at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:5=
23)
> =A0=A0=A0 at org.apache.lucene.store.FSDirectory.copy(FSDirectory.java:45=
0)
> =A0=A0=A0 at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.j=
ava:3019)
> Caused by: java.lang.OutOfMemoryError: Map failed
> =A0=A0=A0 at sun.nio.ch.FileChannelImpl.map0(Native Method)
> =A0=A0=A0 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:767)
> =A0=A0=A0 ... 7 more
>
> I run this on my laptop w/ 4GB RAM. So it's entirely possible there are
> memory issues here. BUT - the segment size is only 300 MB, which is still
> much much less than my machine's RAM.
>
> What worries me is not that particular test - but what will happen if
> someone will try to addIndexes() segments that are 10GB or 100GB or even
> more ... then it really won't matter how much RAM you have. So let's take
> the RAM availability out of the picture.
>
> This API is dangerous, because if someone will try to merge not so large
> segments, on a machine w/ not so much RAM, he'll hit an exception - and i=
t
> didn't happen before 'cause we used byte[] copies (which is slower).
>
> I changed FSDir.copy() code to copy in chunks of 64MB:
>
> =A0=A0=A0=A0=A0=A0=A0 long numWritten =3D 0;
> =A0=A0=A0=A0=A0=A0=A0 long numToWrite =3D input.size();
> =A0=A0=A0=A0=A0=A0=A0 long bufSize =3D 1 << 26;
> =A0=A0=A0=A0=A0=A0=A0 while (numWritten < numToWrite) {
> =A0=A0=A0=A0=A0=A0=A0=A0=A0 numWritten +=3D output.transferFrom(input, nu=
mWritten, bufSize);
> =A0=A0=A0=A0=A0=A0=A0 }
>
> And the process completed successfully.
>
> Obviously, 64MB may be too high for other systems, so I'm thinking we sho=
uld
> make it configurable, but still - chunking, using the same API, succeeds.=
 I
> guess it's just a "not so friendly impl" of Java's FileChannelImpl, but I
> don't know if we can go around it. Maybe we can perf-test and use a small=
er
> chunk size that is safer for all cases (and yields the same performance a=
s
> larger ones) ...
>
> BTW, I don't have FileChannelImpl's source, but Mike found here
> http://www.docjar.com/html/api/sun/nio/ch/FileChannelImpl.java.html. It
> doesn't look like the impl chunks anything ...
>
> What do you think?
>
> Shai

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org