Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 24 Dec 2015 07:06:49 +0000 (UTC)
From: "Kai Zheng (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12836347.1433823079000.1356.1450940809808@Atlassian.JIRA>
In-Reply-To: <JIRA.12836347.1433823079000@Atlassian.JIRA>
References: <JIRA.12836347.1433823079000@Atlassian.JIRA>
 <JIRA.12836347.1433823079167@arcas>
Subject: [jira] [Commented] (HDFS-8562) HDFS Performance is impacted by
 FileInputStream Finalizer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HDFS-8562?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15070=
677#comment-15070677 ]=20

Kai Zheng commented on HDFS-8562:
---------------------------------

I thought this may be pending some while for more reviews as it's dealing w=
ith the performance concern issue not obviously. The javadoc and check styl=
e issues found here would be better to be in sooner on the other hand so I =
would handle them separately in HDFS-12658. Will rebase this after that.

> HDFS Performance is impacted by FileInputStream Finalizer
> ---------------------------------------------------------
>
>                 Key: HDFS-8562
>                 URL: https://issues.apache.org/jira/browse/HDFS-8562
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, performance
>    Affects Versions: 2.5.0
>         Environment: Impact any application that uses HDFS
>            Reporter: Yanping Wang
>         Attachments: HDFS-8562.002b.patch, HDFS-8562.003a.patch, HDFS-856=
2.003b.patch, HDFS-8562.004a.patch, HDFS-8562.01.patch
>
>
> While running HBase using HDFS as datanodes, we noticed excessive high GC=
 pause spikes. For example with jdk8 update 40 and G1 collector, we saw dat=
anode GC pauses spiked toward 160 milliseconds while they should be around =
20 milliseconds.=20
> We tracked down to GC logs and found those long GC pauses were devoted to=
 process high number of final references.=20
> For example, this Young GC:
> 2715.501: [GC pause (G1 Evacuation Pause) (young) 0.1529017 secs]
> 2715.572: [SoftReference, 0 refs, 0.0001034 secs]
> 2715.572: [WeakReference, 0 refs, 0.0000123 secs]
> 2715.572: [FinalReference, 8292 refs, 0.0748194 secs]
> 2715.647: [PhantomReference, 0 refs, 160 refs, 0.0001333 secs]
> 2715.647: [JNI Weak Reference, 0.0000140 secs]
> [Ref Proc: 122.3 ms]
> [Eden: 910.0M(910.0M)->0.0B(911.0M) Survivors: 11.0M->10.0M Heap: 951.1M(=
1536.0M)->40.2M(1536.0M)]
> [Times: user=3D0.47 sys=3D0.01, real=3D0.15 secs]
> This young GC took 152.9 milliseconds STW pause, while spent 122.3 millis=
econds in Ref Proc, which processed 8292 FinalReference in 74.8 millisecond=
s plus some overhead.
> We used JFR and JMAP with Memory Analyzer to track down and found those F=
inalReference were all from FileInputStream.  We checked HDFS code and saw =
the use of the FileInputStream in datanode:
> https://apache.googlesource.com/hadoop-common/+/refs/heads/trunk/hadoop-h=
dfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanod=
e/fsdataset/impl/MappableBlock.java
> {code}
> 1.=09public static MappableBlock load(long length,
> 2.=09FileInputStream blockIn, FileInputStream metaIn,
> 3.=09String blockFileName) throws IOException {
> 4.=09MappableBlock mappableBlock =3D null;
> 5.=09MappedByteBuffer mmap =3D null;
> 6.=09FileChannel blockChannel =3D null;
> 7.=09try {
> 8.=09blockChannel =3D blockIn.getChannel();
> 9.=09if (blockChannel =3D=3D null) {
> 10.=09throw new IOException("Block InputStream has no FileChannel.");
> 11.=09}
> 12.=09mmap =3D blockChannel.map(MapMode.READ_ONLY, 0, length);
> 13.=09NativeIO.POSIX.getCacheManipulator().mlock(blockFileName, mmap, len=
gth);
> 14.=09verifyChecksum(length, metaIn, blockChannel, blockFileName);
> 15.=09mappableBlock =3D new MappableBlock(mmap, length);
> 16.=09} finally {
> 17.=09IOUtils.closeQuietly(blockChannel);
> 18.=09if (mappableBlock =3D=3D null) {
> 19.=09if (mmap !=3D null) {
> 20.=09NativeIO.POSIX.munmap(mmap); // unmapping also unlocks
> 21.=09}
> 22.=09}
> 23.=09}
> 24.=09return mappableBlock;
> 25.=09}
> {code}
> We looked up https://docs.oracle.com/javase/7/docs/api/java/io/FileInputS=
tream.html  and
> http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/23bdcede4e39/src/share/clas=
ses/java/io/FileInputStream.java and noticed FileInputStream relies on the =
Finalizer to release its resource.=20
> When a class that has a finalizer created, an entry for that class instan=
ce is put on a queue in the JVM so the JVM knows it has a finalizer that ne=
eds to be executed.  =20
> The current issue is: even with programmers do call close() after using F=
ileInputStream, its finalize() method will still be called. In other words,=
 still get the side effect of the FinalReference being registered at FileIn=
putStream allocation time, and also reference processing to reclaim the Fin=
alReference during GC (any GC solution has to deal with this).=20
> We can imagine When running industry deployment HDFS, millions of files c=
ould be opened and closed which resulted in a very large number of finalize=
rs being registered and subsequently being executed.  That could cause very=
 long GC pause times.
> We tried to use Files.newInputStream() to replace FileInputStream, but it=
 was clear we could not replace FileInputStream in hdfs/server/datanode/fsd=
ataset/impl/MappableBlock.java=20
> We notified Oracle JVM team of this performance issue that impacting all =
Big Data applications using HDFS. We recommended the proper fix in Java SE =
FileInputStream. Because (1) it is really nothing wrong to use FileInputStr=
eam in above datanode code, (2) as the object with a finalizer is registere=
d with finalizer list within the JVM at object allocation time, if someone =
makes an explicit call to close or free the resources that are to be done i=
n the finalizer, then the finalizer should be pulled off the internal JVM=
=E2=80=99s finalizer list. That will release the JVM from having to treat t=
he object as special because it has a finalizer, i.e. no need for GC to exe=
cute the finalizer as part of Reference Processing. =20
> As the java fix involves both JVM code and Java SE code, it might take ti=
me for the full solution to be available in future JDK releases. We would l=
ike to file his JIRA to notify Big Data, HDFS community to aware this issue=
 while using HDFS and while writing code using FileInputStream=20
> One alternative is to use Files.newInputStream() to substitute FileInputS=
tream if it is possible. File.newInputStream() will give an InputStream and=
 do so in a manner that does not include a finalizer.
> We welcome HDFS community to discuss this issue and see if there are addi=
tional ideas to solve this problem.=20


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)