hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: [Proposal] Pluggable Namespace
Date Tue, 08 Oct 2013 02:40:29 GMT

Seems as a proper time to open a Jira.
Looks to me nobody is objecting to the general idea of AbstractNamesystem.
As usually the details is what finally matters.

FSNamesystem does have a formal interface called Namesystem now, but
it is somewhat arbitrary and probably rudimentary for your purposes.
We should understand what is abstract and what is current implementation
Things like block or inode ids, generation stamps, along with namespace
separated from block management, which currently interleave with each other,
etc., should be considered.

IMO it is better to keep such technical discussion in a jira for future

Your implementation of the namespace, LevelDB effort, Giraffa, our work on
ConsensusNode considered as use cases for AbstractNamesystem can potentially
shape out common requirements for your project.


On Mon, Oct 7, 2013 at 4:50 PM, Milind Bhandarkar <mbhandarkar@gopivotal.com
> wrote:

> Getting back to the technical discussion: based on this proposal email
> that I sent, I came to know, from Sanjay Radia, of a summer intern project
> at Hortonworks that implemented namespace using LevelDB. We discussed it on
> personal email today, and agreed that these two efforts were still
> orthogonal, but will have to be merged because of all the changes that had
> to be made to NN code. Sanjay has promised to post more details on this
> thread. Both the levelDB jira and our jira will be filed soon, and we will
> continue discussions there.
> Now, if I may respond to Doug's email: The LevelDB impl was apparently
> presented at August Yahoo Hadoop meetup, which I did not attend, and since
> I was away from this country, did not even bother to check which talks were
> scheduled. But, I was surprised that no one responded to my proposal
> pointing to that effort. If I had not known about it until much later into
> implementation cycle, it would have been hard for me to account for the
> merge efforts needed.
> How do we keep each other informed what we are planning to do in apache,
> without everyone having to attend non-apache meetups? I think this mailing
> list for proposals is a good choice, but is everyone willing to go on
> record here declaring their plans in advance? In my case, the amount of
> time it took to see whether it was doable, with all the public NS
> modification proposals, to go public with this proposal, was more than
> actually implementing the changes.
> I would seek your inputs to see how we can streamline this process,
> especially for major changes, but perhaps on a separate thread.
> Thanks,
> Milind
> Sent from my iPhone
> >> On Oct 7, 2013, at 12:50, Doug Cutting <cutting@apache.org> wrote:
> >>
> >> On Mon, Oct 7, 2013 at 12:05 PM, Andrew Purtell <apurtell@apache.org>
> wrote:
> >> I recognize some of what we recently experienced on a HDFS matter in
> what
> >> Milind wrote even if this was not the appropriate forum for it. Odd
> mention
> >> of "conspiracy theories" aside, for people who may come to this thread
> >> later, perhaps you can recommend an appropriate public Apache community
> >> forum for discussing such concerns?
> >
> > I'm not exactly sure what "such concerns" are, but I'll guess that
> > you're worried that someone else has plans that conflict with yours
> > yet they're not stating that publicly.  Is that right?
> >
> > We cannot ensure that everyone's contributions are equally loved by
> > everyone else.  Some people's contributions may sometimes conflict
> > with plans of others.  (I have no idea whether that's the case here.)
> > Making a proposal is a good way to inquire about this.  Directly
> > accusing one group of implied opposition is probably
> > counter-productive.
> >
> > We try to ensure that every contribution is given a fair discussion in
> > public and that it is weighed on its technical merit alone.  If a
> > proposal conflicts with some other not-yet-public plan then it's up to
> > the not-public planners to voluntarily make themselves public and try
> > to reach some compromise.  We don't permit vetoes without technical
> > justification, and the benefit of the doubt is generally given to
> > those with code in hand.  So there's little benefit in worrying about
> > what folks aren't saying.
> >
> > The best practice is to first start a discussion around a proposal,
> > often incorporating feedback from the discussion into the proposal.
> > Then, barring fundamental objections to the proposal, contribute code
> > that implements it.  Ideally that code can be developed in public.  It
> > would be poor behavior for someone present at the proposal stage to
> > raise an easily-foreseen fundamental objection in the implementation
> > phase.
> >
> > Does that help?  Can we please return to discussing the proposal?
> >
> > Doug

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message