hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3010) Get performance on HA branch to match trunk
Date Fri, 24 Feb 2012 18:47:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215806#comment-13215806
] 

Todd Lipcon commented on HDFS-3010:
-----------------------------------

There are three possible components to the perf issue, I think:

1) DN now sends RBW replicas to both NNs as soon as a block starts to be created. This adds
3 RPCs to each block creation (though they don't write to the edit logs)
2) When blocks are allocated, we now log the full block list of that file. This creates a
much bigger edit log, so of course takes more time.
3) When HA is enabled, these new edit log entries are fsynced, which makes it even slower.

I'm hoping to set up a cluster to test each of these in isolation by commenting out the related
code from the HA branch and measuring a write benchmark. Once we identify which is the worst
issue we can tackle it.
                
> Get performance on HA branch to match trunk
> -------------------------------------------
>
>                 Key: HDFS-3010
>                 URL: https://issues.apache.org/jira/browse/HDFS-3010
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>
> As described in [this comment|https://issues.apache.org/jira/browse/HDFS-1623?focusedCommentId=13215309&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13215309]
the performance of the HA branch for writes is significantly reduced compared to trunk. We
need to dig a bit and optimize whatever it is that's hurting us in order to get back to the
same performance numbers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message