hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: Merging Namenode Federation feature (HDFS-1052) to trunk
Date Tue, 15 Mar 2011 01:12:30 GMT
Dhruba, good you are speaking up for federation.
I consider it important as it means more support for the feature in the

The purpose of my reply was to get this discussion going, as I found Allens
question unanswered for 2 weeks.
The concern he has seems legitimate to me. If ops think federation will
"make running a grid much much harder" I want to know why and how much
Because cluster "manageability" is claimed as one of the objectives of

I sure am well familiar with the design being a part of it for a while.
And all my concerns have been articulated and well known. Though not all of
them are addressed.

The way I see it now, Federation introduces
- lots of code complexity to the system
- harder manageability, according to Allen
- potential performance degradation (tbd)
And the main question for those 95% of users, who don't run large clusters
don't want to place all their compute resources in one data center, is what
is the advantage in supporting it?

Performance-wise there 2 main aspects:
- Does federation give me the same cluster performance if I don't federate?
- If I federate how much more throughput can I get?


On Mon, Mar 14, 2011 at 10:43 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> Hi folks,
> The design for the federation work has been a published and there is a very
> well-written design document. It explains the pros-and-cons of each design
> point. It would be nice if more people can review this document and provide
> comments on how to make it better. The implementation is in progress but
> that does not mean that the
> "design-is-cast-in-stone-and-cannot-be-enhanced".
> Allen: can you pl describe what you mean by  "It sounds like merging into
> trunk is extremely premature". If we can make all unit tests pass
> successfully on the branch, then do you think we should merge that branch
> into the trunk?
> Konstantin: I agree that federation introduces new code complexity. But it
> is a fact that introducing a new heavy-weight feature will add complexity.
> If you have a different proposal (and implementation) to scale namenode,
> please share it with us and we can then evaluate these designs in terms on
> complexity/feature. If you have questions about certain issues in the
> design, it would be great if you can ask them now. Hopefully, the folks
> doing the implementation can then provide you performance numbers to
> alleviate your concerns.
> From that way I look at it, I think the federation-feature is a huge
> positive step in the right direction.
> thanks,
> dhruba
> On Mon, Mar 14, 2011 at 10:28 AM, Konstantin Shvachko <
> shv.hadoop@gmail.com> wrote:
>> Allen is right.
>> This is a huge new feature with 86 jiras already filed, which
>> substantially
>> increases the complexity of the code base.
>> Having an in-depth motivation and benchmarking will be needed before the
>> community decides on adopting it for support.
>> Thanks,
>> --Konstantin
>> On Sat, Mar 12, 2011 at 8:43 AM, Allen Wittenauer
>> <awittenauer@linkedin.com>wrote:
>> >
>> > On Mar 3, 2011, at 2:41 PM, Suresh Srinivas wrote:
>> >
>> > > We have started pushing changes for namenode federation in to the
>> feature
>> > branch HDFS-1052. The work items are created as subtask of the jira
>> > HDFS-1052 and are based on the design document published in the same
>> jira.
>> > By the end of this week, we will complete pushing the changes to
>> HDFS-1052
>> > branch. Though the changes in these jiras are already committed, please
>> do
>> > provide your feedback on either HDFS-1052 or its subtasks. New items
>> that
>> > come out of the feedback will be addressed in new jiras.
>> >
>> > >
>> > > Current status of the development:
>> > > # The testing of this feature is underway. Most of the basic
>> > functionality has been tested both for a single namenode cluster (for
>> > backward compatibility) and with multiple namenodes.
>> > > # All the existing tests and newly added tests pass (same as trunk).
>> > >
>> > > We plan on merging this branch to trunk after a week or two. This will
>> > help us continue make future changes on the trunk. I will send an
>> > announcement before merging the federation branch into trunk.
>> > >
>> >
>> >        It sounds like merging into trunk is extremely premature.  That
>> > said, I'm still trying to understand the why's around this.
>> >
>> >        To me, this series of changes looks like it is going to make
>> running
>> > a grid much much harder for very little benefit.  In particular, I don't
>> see
>> > the difference between running multiple NN/DN combinations verses
>> running
>> > federation, especially with client side mount tables in play.
>> >
>> >
> --
> Connect to me at http://www.facebook.com/dhruba

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message