hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2142) Namenode in trunk has much slower performance than Namenode in MR-279 branch
Date Tue, 12 Jul 2011 20:23:00 GMT
Namenode in trunk has much slower performance than Namenode in MR-279 branch

                 Key: HDFS-2142
                 URL: https://issues.apache.org/jira/browse/HDFS-2142
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: name-node
    Affects Versions: 0.23.0
            Reporter: Eric Payne

I am measureing the performance of the namenode by running the org.apache.hadoop.fs.loadGenerator.LoadGenerator
application. This application shows there is a very large slowdown in the processing of opens,
writes, closes, and operations per second in the trunk when compared to the MR-279 branch

There have been some race conditions and locking issues fixed in trunk, which is a very good
thing because these race conditions were causing the namenode to crash under load conditions
(see HDFS:1257). However, the slowdown to the namenode is considerable.

I am still trying to verify which changes caused the slowdown. It was originally suggested
that the HDFS:988 may have caused the slowdown, but I don't think it was the culprit. I have
checked out and built from SVN 3 revisions previous to HDFS988 and they all have about the
same performance.

Here is my environment:
Host0: namenode daemon
Host1-9: simulate many datanodes using org.apache.hadoop.hdfs.DataNodeCluster
LoadGenerator output on MR-279 branch:
Average open execution time: 1.8496516782773909ms
Average deletion execution time: 2.956340167046317ms
Average create execution time: 3.725259427992913ms
Average write_close execution time: 11.151860288534548ms
Average operations per second: 1053.3666666666666ops/s

LoadGenerator output on trunk:
Average open execution time: 28.603515625ms
Average deletion execution time: 32.20792079207921ms
Average create execution time: 32.37326732673267ms
Average write_close execution time: 82.84752475247525ms
Average operations per second: 135.13333333333333ops/s

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message