hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hash-X <...@git.apache.org>
Subject [GitHub] hadoop pull request: AltFileInputStream.java replace FileInputStre...
Date Thu, 05 Nov 2015 03:26:06 GMT
GitHub user hash-X opened a pull request:


    AltFileInputStream.java replace FileInputStream.java in apache/hadoop/HDFS

    A brief description
    Long Stop-The-World GC pauses due to Final Reference processing are observed.
    So, Where are those Final Reference come from ?
    1 : `Finalizer`
    2 : `FileInputStream`
    How to solve this problem ?
    Here is the detailed description,and I give a solution on this.
    FileInputStream have a method of finalize , and it can cause GC pause for a long time.In
our test,G1 as our GC. So,in AltFileInputStream , no finalize. A new design for a inputstream
use in windows and non-windows.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/hash-X/hadoop AltFileInputStream

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #42
commit 8d64ef0feb8c8d8f5d5823ccaa428a1b58f6fd04
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-19T09:50:19Z

    Add some code.

commit 3ccf4c70c40cf1ba921d76b949317b5fd6752e3c
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-19T09:56:49Z

    I cannot replace FileInputStream to NewFileInputStream in a casual way,cause the act
    of change can damage other part of the HDFS.For example,When I test my code
    using a Single Node (psedo-distributed) Cluster."Failed to load an FSImage
    file." will happen when I start HDFS Daemons.At start,I replace
    many FileInputStream which happend as an arg or constructor to
    NewFileInputStream,but it seems like wrong.So,I have to do this in another

commit 4da55130586ee9803a09162f7e2482b533aa12d9
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-19T10:30:11Z

    Replace FIS to NFIS( NewFileInputStream ) is not recommend I
    think,although there is a man named Alan Bateman from
    suggest that.But test shows it is not good.Some problem may happen.And
    these test consume so long time.Every time I change the source code,I need to
    build the whole project (maybe it is not needed).But I install the new
    version hadoop on my computer.So,build the whole project is needed.Maybe
    should have a good way to do it I think.

commit 06b1509e0ad6dd74cf7c903e6ed6f2ec74d9b341
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-19T11:06:37Z

    Replace FIS to NFIS,If test success,just do these first.It is not as
    simple as that.

commit 2a79cd9c3b012556af7db5bdbf96663a1c30dcc4
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T02:36:55Z

    Add a LOG info in DataXceiver for test.

commit 436c998ae21b3fe843b2d5ba6506e37ff2a34ab2
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T06:01:41Z

    Rename NewFileInputStream to AltFileInputStream.

commit 14de2788ea2407c6ee252a69cfd3b4f6132c6faa
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T06:16:32Z

    replace License header to Apache.

commit 387f7624a96716abef2062986f05523199e1927e
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T07:16:25Z

    Remove open method in AltFileInputStream.java.

commit 52b029fac56bc054add1eac836e6cf71a0735304
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T10:14:09Z

    Performance between AltFileInputStream and FileInputStream is not do
    from this commit.Important question I think whether
    AltFileInputStream could convert
    to FileInputStream safely.I define a frame plan to do it.But I don't know is
    this correct for the problem ? In HDFS code,compulsory conversion to
    FileInputStream is happend everywhere.

commit e76d5eb4bf0145a4b28c581ecec07dcee7bae4e5
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T13:11:24Z

    I think the compulsory conversion is safety,cause the AltFileInputSteam is
    the subclass of the InputStream.In the previous version of the HDFS,the
    convert to FileInputStream I see is safety cause these method return
    InputStream which is the superclass of the FileInputStream.In my version
    of HDFS,InputStream is also the superclass of the
    AltFileInputStream.So,AltFileInputStream is also a InputStream just like
    the FileInputStream is a InputStream too.So,I think it is
    safety.Everyone agree ? If not,please give your opinion and tell me What's
    wrong with that ? thank you.

commit 959e91c05c11cc1445513d36ec083707f0bba4e1
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T14:08:51Z

    channel.close() is not needed.Because closeing the stream will in turn
    cause the channel to be closed.

commit aa7f82efb29d6ff457dcf6e5b2a74af663682106
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-20T15:57:36Z

    Delete HDFS-8562.patch
    old patch.

commit 5e8e15cecb6d159706270dafd6e71dc9816abf19
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-21T02:56:34Z

    Revice AltFileInputStream.java

commit 1d0e7fb8f9cca59480d4a79d62cdf005a804912f
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-22T10:32:45Z

    Make a test.

commit 5daf8730dbc3e94fe766acf83b3ace39e60bc730
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-22T14:32:02Z

    update AltFileInputStream.java

commit f58060ce118d6991824c9c9f622fe54776c2d32d
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-23T06:22:49Z

    Test class of AltFileInputStream.java

commit 76673377a73624ff60322c0ae4cdd61701ea79fc
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-23T08:05:52Z

    new test file.

commit fea1be8a7b21d15b0b0020c12c313eff108acd56
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-23T13:48:51Z

    The latest AltFileInputStream.java

commit 3ad1b9b25f54e68d6816dfbc477a3c4221f445bd
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-23T14:03:03Z

    The latest new file of AFIS

commit 8805c8bd87985d840eb534cad6c5e891b7f6ca41
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T00:59:44Z

    Latest file of AltFileInputStream.java

commit 5cdabc49963652e7bb65aa00a448a5c06793b421
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T01:47:16Z


commit 221d27dbb908cc606a495cd298a7717ad65733aa
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T02:01:17Z

    Soga ....

commit 69d147d98f83ee564df73ae7f6ac6d4bdcf3c816
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T03:02:24Z

    a patch

commit a6de350f5240554015a54e8a07e399aaf6e89b61
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T04:39:21Z

    Latest file.

commit 0860e5da144ea8bfe51d378b212921f365904497
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T04:50:38Z

    update AltFileInputStream.

commit a566a304650cc4b8b3b3e5f700c564169bfe6d5b
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-24T08:22:27Z


commit 02d53c9227ad124290042129cdd872b48c24dd78
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-27T08:21:10Z

    update some code.

commit 090a4c8dd1eee4e01b791e623775b2074f9e6afb
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-27T09:06:32Z

    update some code.

commit 63047775b3b528bf1e831f6c5baa2b8ccdba0f24
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-07-27T09:19:45Z

    update some code.

commit 6d3deaa377b85a3f24bd8d7e7a4d7682e8949ec1
Author: zhangminglei <minglei.l.zhang@intel.com>
Date:   2015-09-25T05:06:17Z

    Add some info


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message