hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1052) HDFS scalability with multiple namenodes
Date Fri, 20 Aug 2010 23:17:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900910#action_12900910

Suresh Srinivas commented on HDFS-1052:

ryan, this is discussed in the proposal already. Let me summarize:
# Increasing the namenode heap does not increase the namenode throughput
# Currently NN takes 30 mins to startup with 50G heap. The startup time would go to 2.5 hrs.
There are couple of jiras improving the NN startup time. Even with that, start up time would
be > 1 hr for such a large heap.
# While debugging memory leaks in NN, I could not get lot of tools to work with the heap dump
of 40G, especially jhat. Not sure how well the tools can support 250G heap dump.
# This solution does not work for installation where the NN needs to support more 4x scaling.
This is needed in clusters that might want to store smaller files instead of depending on
large files to reduce object count.

The solution proposed here does not preclude one from using a single namenode and vertically
scaling it.

I am also curious about your experience and challenges of running a namenode with such large
heap. We could have that discussion offline.

> HDFS scalability with multiple namenodes
> ----------------------------------------
>                 Key: HDFS-1052
>                 URL: https://issues.apache.org/jira/browse/HDFS-1052
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: Block pool proposal.pdf, Mulitple Namespaces5.pdf
> HDFS currently uses a single namenode that limits scalability of the cluster. This jira
proposes an architecture to scale the nameservice horizontally using multiple namenodes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message