hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
Date Wed, 29 Feb 2012 01:36:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218813#comment-13218813
] 

Todd Lipcon commented on HDFS-1623:
-----------------------------------

With HDFS-3020, HDFS-3023, HDFS-3024, HDFS-3025, I got the following timings:

{code}
teragen-4MB-block-ha-off-1.txt:         Total time spent by all maps in occupied slots (ms)=255005244
teragen-4MB-block-ha-off-2.txt:         Total time spent by all maps in occupied slots (ms)=248092620
teragen-4MB-block-ha-off-3.txt:         Total time spent by all maps in occupied slots (ms)=256926353
teragen-4MB-block-ha-off-4.txt:         Total time spent by all maps in occupied slots (ms)=244320729
teragen-4MB-block-ha-off-5.txt:         Total time spent by all maps in occupied slots (ms)=248901067
teragen-4MB-block-ha-off-6.txt:         Total time spent by all maps in occupied slots (ms)=234409970
teragen-4MB-block-ha-off-7.txt:         Total time spent by all maps in occupied slots (ms)=224624077
teragen-4MB-block-ha-off-8.txt:         Total time spent by all maps in occupied slots (ms)=235166437
----
teragen-4MB-block-trunk-1.txt:          Total time spent by all maps in occupied slots (ms)=247575318
teragen-4MB-block-trunk-2.txt:          Total time spent by all maps in occupied slots (ms)=234090512
teragen-4MB-block-trunk-3.txt:          Total time spent by all maps in occupied slots (ms)=241264032
teragen-4MB-block-trunk-4.txt:          Total time spent by all maps in occupied slots (ms)=242941073
teragen-4MB-block-trunk-5.txt:          Total time spent by all maps in occupied slots (ms)=236123386
teragen-4MB-block-trunk-6.txt:          Total time spent by all maps in occupied slots (ms)=243662148
teragen-4MB-block-trunk-7.txt:          Total time spent by all maps in occupied slots (ms)=240128084
teragen-4MB-block-trunk-8.txt:          Total time spent by all maps in occupied slots (ms)=220212020
{code}

I ran a t-test which says that the difference in means isn't statistically significant. I'm
also running the 256M-block teragen just to be safe. It's not complete yet but so far the
results look good.

The optimizations also reduced the edit log size for the 4MB-block by a factor of two.

So I think once these above JIRAs are committed, we should be fine to merge to trunk. I'll
also continue to work on the performance with HA on, but the important issue for merge is
to make sure we don't regress the non-HA case.
                
> High Availability Framework for HDFS NN
> ---------------------------------------
>
>                 Key: HDFS-1623
>                 URL: https://issues.apache.org/jira/browse/HDFS-1623
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: HA-tests.pdf, HDFS-1623.trunk.patch, HDFS-High-Availability.pdf,
NameNode HA_v2.pdf, NameNode HA_v2_1.pdf, Namenode HA Framework.pdf, dfsio-results.tsv, ha-testplan.pdf,
ha-testplan.tex
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message