hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mawata <chris.maw...@gmail.com>
Subject Re: HDFS / Federated HDFS - Doubts
Date Wed, 02 Oct 2013 18:43:37 GMT
One more thing, Krishna,  when using JounalNodes as opposed to the 
native file system for the metadata storage  you do get replication.
Chris


On 10/2/2013 12:52 AM, Krishna Kumaar Natarajan wrote:
> Hi All,
>
> While trying to understand federated HDFS in detail I had few doubts 
> and listing them down for your help.
>
>  1. In case of *_HDFS(without HDFS federation)_*, the metadata or the
>     data about the blocks belonging to the files in HDFS is maintained
>     in the main memory of the name node or it is stored on permanent
>     storage of the namenode and is brought in the main memory on
>     demand basis ?[Krishna] Based on my understanding, I assume the
>     entire metadata is in main memory which is an issue by itself.
>     Please correct me if my understanding is wrong.
>  2. In case of*_federated HDFS_*, the metadata or the data about the
>     blocks belonging to files in a particular namespace is maintained
>     in the main memory of the namenode or it is stored on the
>     permanent storage of the namenode and is brought in the main
>     memory on demand basis ?
>  3. Are the metadata information stored in separate cluster
>     nodes(block management layer separation) as discussed in Appendix
>     B of this document
>     ?https://issues.apache.org/jira/secure/attachment/12453067/high-level-design.pdf
>  4. I would like to know if the following proposals are already
>     implemented in federated HDFS.
>     (http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability
slide-17)
>       * Separation of namespace and block management layers (same as qn.3)
>       * Partial namespace in memory for further scalability
>       * Move partial namespace from one namenode to another
>
> Thanks,
> Krishna


Mime
View raw message