hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Kumaar Natarajan <natar...@umn.edu>
Subject HDFS / Federated HDFS - Doubts
Date Wed, 02 Oct 2013 04:52:59 GMT
Hi All,

While trying to understand federated HDFS in detail I had few doubts and
listing them down for your help.

   1. In case of *HDFS(without HDFS federation)*, the metadata or the data
   about the blocks belonging to the files in HDFS is maintained in the main
   memory of the name node or it is stored on permanent storage of the
   namenode and is brought in the main memory on demand basis ? [Krishna]
   Based on my understanding, I assume the entire metadata is in main memory
   which is an issue by itself. Please correct me if my understanding is wrong.
   2. In case of* federated HDFS*, the metadata or the data about the
   blocks belonging to files in a particular namespace is maintained in the
   main memory of the namenode or it is stored on the permanent storage of the
   namenode and is brought in the main memory on demand basis ?
   3. Are the metadata information stored in separate cluster nodes(block
   management layer separation) as discussed in Appendix B of this document ?
   https://issues.apache.org/jira/secure/attachment/12453067/high-level-design.pdf
   4. I would like to know if the following proposals are already
   implemented in federated HDFS. (
   http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability
    slide-17)
   - Separation of namespace and block management layers (same as qn.3)
      - Partial namespace in memory for further scalability
      - Move partial namespace from one namenode to another

Thanks,
Krishna

Mime
View raw message