hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
Date Fri, 24 Feb 2012 02:07:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215309#comment-13215309

Todd Lipcon commented on HDFS-1623:

I've completed a round of preliminary performance testing on a 100-node cluster, each with
48G RAM, gigabit ethernet, 6 disks, dual quad core with hyperthreading.

The three builds were trunk (rev 5b8bdf3c11fc69a9076e2b1e3385ecd179975c7d), HA branch (tip),
and the same HA branch but with HA turned off.

The HA NN is configured to log to an NFS mount, which unfortunately is just a normal Linux
NFS box in this case, in the same rack. It was also unfortunately running a DN, though I didn't
realize it until I was halfway through the benchmarks, and didn't want to start over from

I ran the TestDFSIO benchmarks as suggested by Konstantin, first with 95 files and then with
380 files. Each file is 5GB. I will attach the results as a TSV file momentarily. The overall
summary is that there seems to be a ~4% hit for turning on HA for the write benchmark. The
read benchmark has too high variability to really be sure. Even the write one doesn't seem
that consistent, given that the HA branch, when HA was disabled, actually went some 2.5% faster
than trunk. (these numbers from the 380-file case)

I also ran teragen in two different scenarios. The first scenario is a realistic workload
(256M blocks):
|| Build || Runtime || Slot time ||
|| HA On ||  8m5s | 3682m |
|| HA Off || 8m12s | 3756m |
|| Trunk || 7m10s | 3163m |

Here there seems to be a bad performance degredation from the HA branch (about 20%).

The second workload was to try to stress the system by setting block size to 4MB (resulting
in ~800-1000 block allocations/second):

|| Build || Runtime || Slot time || Edit log size ||
|| HA On || 12m24s |  6655m | 1.2GB |
|| HA Off || 8m40s | 4469m | 1.2GB |
|| Trunk || 7m4s | 3375m | 6.2MB |

Note here the much bigger degredation. I also included the size of the edit logs in this later

I'm pretty confident from looking at jstacks while this was running that the bad performance
is due to the new "persistBlocks" calls done in the HA branch. We used to be sloppy about
persisting blocks, whereas now we actually write down all of the block allocations as they
proceed. The 4MB block case was much worse, since each file being written by teragen consisted
of 423 blocks. The persist blocks calls towards the end of the file were logging several KB
worth of data, and this resulted in a very large edit log as you can see in the table above.
The difference between HA-on and HA-off is that the HA-on mode actually fsyncs all of these
block allocations.

So before we merge I think we should do a bit of optimization in this area. I will file a
JIRA this evening or tomorrow with a couple of easy wins.
> High Availability Framework for HDFS NN
> ---------------------------------------
>                 Key: HDFS-1623
>                 URL: https://issues.apache.org/jira/browse/HDFS-1623
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: HA-tests.pdf, HDFS-1623.trunk.patch, HDFS-High-Availability.pdf,
NameNode HA_v2.pdf, NameNode HA_v2_1.pdf, Namenode HA Framework.pdf, ha-testplan.pdf, ha-testplan.tex

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message