hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-2142) Namenode in trunk has much slower performance than Namenode in MR-279 branch
Date Fri, 15 May 2015 17:53:02 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Allen Wittenauer resolved HDFS-2142.
    Resolution: Cannot Reproduce


> Namenode in trunk has much slower performance than Namenode in MR-279 branch
> ----------------------------------------------------------------------------
>                 Key: HDFS-2142
>                 URL: https://issues.apache.org/jira/browse/HDFS-2142
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.23.0
>            Reporter: Eric Payne
> I am measureing the performance of the namenode by running the org.apache.hadoop.fs.loadGenerator.LoadGenerator
application. This application shows there is a very large slowdown in the processing of opens,
writes, closes, and operations per second in the trunk when compared to the MR-279 branch
> There have been some race conditions and locking issues fixed in trunk, which is a very
good thing because these race conditions were causing the namenode to crash under load conditions
(see HDFS:1257). However, the slowdown to the namenode is considerable.
> I am still trying to verify which changes caused the slowdown. It was originally suggested
that the HDFS:988 may have caused the slowdown, but I don't think it was the culprit. I have
checked out and built from SVN 3 revisions previous to HDFS988 and they all have about the
same performance.
> Here is my environment:
> Host0: namenode daemon
> Host1-9: simulate many datanodes using org.apache.hadoop.hdfs.DataNodeCluster
> LoadGenerator output on MR-279 branch:
> Average open execution time: 1.8496516782773909ms
> Average deletion execution time: 2.956340167046317ms
> Average create execution time: 3.725259427992913ms
> Average write_close execution time: 11.151860288534548ms
> Average operations per second: 1053.3666666666666ops/s
> LoadGenerator output on trunk:
> Average open execution time: 28.603515625ms
> Average deletion execution time: 32.20792079207921ms
> Average create execution time: 32.37326732673267ms
> Average write_close execution time: 82.84752475247525ms
> Average operations per second: 135.13333333333333ops/s

This message was sent by Atlassian JIRA

View raw message