hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From suresh srinivas <srini30...@gmail.com>
Subject Re: Merging Namenode Federation feature (HDFS-1052) to trunk
Date Wed, 16 Mar 2011 23:54:08 GMT
That is a different motivation. The document talks about why you should use
> federation. I am asking about motivation of supporting the code base while
> not using it. At least this is how understand Allen's question and some of
> my colleagues'.

Namenode code is not changed at all. Datanode code changes to add the notion
of block pool and a thread per NN. For a single NN, datanode is equivalent
to the current datanode. If you argue that there should not be any code
change - not sure how features like this can be added to HDFS. There is no
change from user perspective and performance of the system. No additional
complexity from the existing system.

> If you could put some numbers in the jira for the reference.
Will do.

> Also it is interesting to know whether there is a benefit in splitting
> the namespace. Can I e.g. do more getBlockLocations per second?
> This is one of the aspects of scaling, right?

I do not understand your question. This feature does not scale
getBlockLocations per second for a single NN. When you use many NNs, total
requests per second does scale for the entire cluster.

> As we developed this feature, some significant improvements have been made
> to the system - fast snapshots (snapshot time down from 1hr 45 mins to 1
> min!), fast startup, cleanup of storage, fixing multi threading issues in
> several places, decommissioning improvements etc.

> This is a valid concern. Hence the single namenode configuration that most
> > installations run today, will run as is. We put a lot of development and
> > testing effort to ensure this.
> >
> I don't know what you mean by "as is". My experience with this word in real
> estate tells me it can be anything.

I used the word with following meaning:
— *as is*
*:* in the presently existing condition without modification

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message