hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chao Huang <chaomhu...@gmail.com>
Subject Re: New bie questions re: namenode and masters
Date Thu, 24 May 2012 13:21:59 GMT
Hi Ravi,

Thanks for your reply. It's very helpful.

No, I was talking about this:
http://hadoop.apache.org/common/docs/r1.0.3/hdfs_design.html (hadoop
version:  r1.0.3). It says:

HDFS has a master/slave architecture. An HDFS cluster consists of a single
NameNode, a master server that manages the file system namespace and
regulates access to files by clients. In addition, there are a number of
DataNodes, usually one per node in the cluster, which manage storage
attached to the nodes that they run on.

This part confuses me, and made me think that the NameNode and the master
server should run on two exclusive nodes. Now according to your
explanation, I think masters mean the NameNodes, while slaves mean the
DataNodes. I hope this is correct now.  :)

Best regards,

On Thu, May 24, 2012 at 7:58 PM, Ravi Prakash <ravihadoop@gmail.com> wrote:

> Hi Chao,
> What documentation are you reading? This is pretty accurate :
> http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_design.html
> The NameNode is indeed responsible for the metadata. And all the datanodes
> report to the NameNode (so they are all slaves). You are right, the data
> blocks are stored on the DataNodes. Perhaps I am lacking knowledge of the
> history, but as of now there's no "master server". All read write requests
> on files are directed at the Namenode from where they get redirected to the
> appropriate DataNode holding the block.
> So your configuration for replication factor 3 would look like:
> in conf/core-site.xml:
>     fs.default.name = hdfs://machineAAA:54321/
> in conf/slaves:
>     machineBBB
>     machineCCC
>     machineDDD
>     machineEEE
>     ....possibly a lot more
> Hope this helps
> Ravi
> On Thu, May 24, 2012 at 6:30 AM, Chao Huang <chaomhuang@gmail.com> wrote:
>> Hello experts,
>> I'm new to hdfs/hadoop.  After reading the hdfs documents, I'm getting
>> confused by the differences between a namenode and a master server.  It's
>> my understanding that the namenode is responsible for managing metadata,
>> while the master-replica group (which is comprised by a number of
>> datanodes) stores the actual data blocks.  In the master-replica group, the
>> master server accepts read/write requests, and load balances (or routes)
>> read requests to the appropriate replica. In other words, we should
>> configure the namenode and master server on two different physical machines
>> in a production environment, right?  Is this a correct assumption?
>> One other question about HDFS cluster setup:
>> - requirements:  one namenode, replication factor = 3, in a production
>> environment.
>> how would the topology look like?  Can I configure as follows?
>> in conf/core-site.xml:
>>     fs.default.name = hdfs://machineAAA:54321/
>> in conf/masters:
>>     machineBBB
>> in conf/slaves:
>>     machineCCC
>>     machineDDD
>> Can someone please confirm and/or comment?
>> Sorry for my new bie questions. Thanks for the help.
>> Chao

View raw message