hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: [VOTE] Merge HDFS-4949 to trunk
Date Thu, 24 Oct 2013 23:02:23 GMT
On Thu, Oct 24, 2013 at 1:45 PM, Chris Nauroth <cnauroth@hortonworks.com> wrote:
> Hi Andrew,
>
> I've come to the conclusion that I'm very confused about merge votes.  :-)
>  It's not just about HDFS-4949.  I'm confused about all merge votes.
>  Rather than muddy the waters here, I've started a separate discussion on
> common-dev.
>
> I do agree with the general plan outlined here, and I will comment directly
> on the HDFS-4949 jira with a binding +1 when I see that we've completed
> that plan.

Thanks, Chris.  Andrew posted a merge patch to HDFS-4949.

We're happy that this code is getting closer to getting into trunk,
since it will make it easier to integrate with the other features in
trunk (like HDFS-2832).  There are still some follow-up tasks, but we
feel that it's easier to do those in trunk.

I'm going to update the design doc in just a moment so be sure to
check it out.  Are there any other things we should do today prior to
merging?

Colin


>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Oct 23, 2013 at 2:18 PM, Andrew Wang <andrew.wang@cloudera.com>wrote:
>
>> Hey Chris,
>>
>> Right now we're on track to have all of those things done by tomorrow.
>> Since the remaining issues are either not technical or do not involve major
>> changes, I was hoping we could +1 this merge vote in the spirit of "+1
>> pending jenkins". We've gotten clean unit test runs on upstream Jenkins as
>> well, so the only fixups we should need for test-patch.sh are findbugs and
>> javac (which are normally pretty trivial to clean up). Of course, all of
>> your listed prereqs and test-patch would be taken care of before actually
>> merging to trunk.
>>
>> So, we can reset the vote if you feel strongly about this, but it seems
>> like the only real result will be delaying the merge by a week.
>>
>> Thanks,
>> Andrew
>>
>>
>> On Wed, Oct 23, 2013 at 1:03 PM, Chris Nauroth <cnauroth@hortonworks.com
>> >wrote:
>>
>> > I've received some feedback that we haven't handled this merge vote the
>> > same as other comparable merge votes, and that the vote should be reset
>> > because of this.
>> >
>> > The recent custom is that we only call for the merge vote after all
>> > pre-requisites have been satisfied.  This would include committing to the
>> > feature branch all patches that the devs deem necessary before the code
>> > lands in trunk, posting a test plan, posting an updated design doc in
>> case
>> > implementation choices diverged from the original design doc, and
>> getting a
>> > good test-patch run from Jenkins on the merge patch.  This was the
>> process
>> > followed for other recent major features like HDFS-2802 (snapshots),
>> > HDFS-347 (short-circuit reads via sharing file descriptors), and
>> > HADOOP-8562 (Windows compatibility).  In this thread, we've diverged from
>> > that process by calling for a vote on a branch that hasn't yet completed
>> > the pre-requisites and stating a plan for work to be done before the
>> merge.
>> >
>> > I still support this work, but can we please restart the vote after the
>> > pre-requisites have landed in the branch?
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Fri, Oct 18, 2013 at 1:37 PM, Chris Nauroth <cnauroth@hortonworks.com
>> > >wrote:
>> >
>> > > +1
>> > >
>> > > Sounds great!
>> > >
>> > > Regarding testing caching+federation, this is another thing that I had
>> > > intended to pick up as part of HDFS-5149.  I'm not sure if I can get
>> this
>> > > done in the next 7 days, so I'll keep you posted.
>> > >
>> > > Chris Nauroth
>> > > Hortonworks
>> > > http://hortonworks.com/
>> > >
>> > >
>> > >
>> > > On Fri, Oct 18, 2013 at 11:15 AM, Colin McCabe <cmccabe@alumni.cmu.edu
>> > >wrote:
>> > >
>> > >> Hi Chris,
>> > >>
>> > >> I think it's feasible to complete those tasks in the next 7 days.
>> > >> Andrew is on HDFS-5386.
>> > >>
>> > >> The test plan document is a great idea.  We'll try to get that up
>> > >> early next week.  We have a lot of unit tests now, clearly, but some
>> > >> manual testing is important too.
>> > >>
>> > >> If we discover any issues during testing, then we can push out the
>> > >> merge timeframe.  For example, one area that probably needs more
>> > >> testing is caching+federation.
>> > >>
>> > >> I would like to get HDFS-5378 and HDFS-5366 in as well.
>> > >>
>> > >> The other subtasks are "nice to have" but not really critical, and
I
>> > >> think it would be just as easy to do them in trunk.  We're hoping that
>> > >> having this in trunk will make it easier for us to collaborate on
>> > >> HDFS-2832 and other ongoing work.
>> > >>
>> > >> > Also, I want to confirm that this vote only covers trunk.
>> > >> > I don't see branch-2 mentioned, so I assume that we're
>> > >> > not voting on merge to branch-2 yet.
>> > >>
>> > >> Yeah, this vote is only to merge to trunk.
>> > >>
>> > >> cheers.
>> > >> Colin
>> > >>
>> > >> On Fri, Oct 18, 2013 at 10:48 AM, Chris Nauroth
>> > >> <cnauroth@hortonworks.com> wrote:
>> > >> > I agree that the code has reached a stable point.  Colin and Andrew,
>> > >> thank
>> > >> > you for your contributions and collaboration.
>> > >> >
>> > >> > Throughout development, I've watched the feature grow by running
>> daily
>> > >> > builds in a pseudo-distributed deployment.  As of this week, the
>> full
>> > >> > feature set is working end-to-end.  I also think we've reached
a
>> point
>> > >> of
>> > >> > API stability for clients who want to control caching
>> > programmatically.
>> > >> >
>> > >> > There are several things that I'd like to see completed before
the
>> > >> merge as
>> > >> > pre-requisites:
>> > >> >
>> > >> > - HDFS-5203: Concurrent clients that add a cache directive on
the
>> same
>> > >> path
>> > >> > may prematurely uncache from each other.
>> > >> > - HDFS-5385: Caching RPCs are AtMostOnce, but do not persist client
>> ID
>> > >> and
>> > >> > call ID to edit log.
>> > >> > - HDFS-5386: Add feature documentation for datanode caching.
>> > >> > - Standard clean-ups to satisfy Jenkins pre-commit on the merge
>> patch.
>> > >> >  (For example, I know we've introduced some Javadoc warnings.)
>> > >> > - Full test suite run on Windows.  (The feature is not yet
>> implemented
>> > >> on
>> > >> > Windows.  This is just intended to catch regressions.)
>> > >> > - Test plan posted to HDFS-4949, similar in scope to the snapshot
>> test
>> > >> plan
>> > >> > that was posted to HDFS-2802.  For my own part, I've run the new
>> unit
>> > >> > tests, and I've tested end-to-end in a pseudo-distributed
>> deployment.
>> > >>  It's
>> > >> > unlikely that I'll get a chance to test fully distributed before
the
>> > >> vote
>> > >> > closes, so I'm curious to hear if you've done this on your side
yet.
>> > >> >
>> > >> > Also, I want to confirm that this vote only covers trunk.  I don't
>> see
>> > >> > branch-2 mentioned, so I assume that we're not voting on merge
to
>> > >> branch-2
>> > >> > yet.
>> > >> >
>> > >> > Before I cast my vote, can you please discuss whether or not it's
>> > >> feasible
>> > >> > to complete all of the above in the next 7 days?  For the issues
>> > >> assigned
>> > >> > to me, I do expect to complete them.
>> > >> >
>> > >> > Thanks again for all of your hard work!
>> > >> >
>> > >> > Chris Nauroth
>> > >> > Hortonworks
>> > >> > http://hortonworks.com/
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Thu, Oct 17, 2013 at 3:07 PM, Colin McCabe <
>> cmccabe@alumni.cmu.edu
>> > >> >wrote:
>> > >> >
>> > >> >> +1.  Thanks, guys.
>> > >> >>
>> > >> >> best,
>> > >> >> Colin
>> > >> >>
>> > >> >> On Thu, Oct 17, 2013 at 3:01 PM, Andrew Wang <
>> > andrew.wang@cloudera.com
>> > >> >
>> > >> >> wrote:
>> > >> >> > Hello all,
>> > >> >> >
>> > >> >> > I'd like to call a vote to merge the HDFS-4949 branch
(in-memory
>> > >> caching)
>> > >> >> > to trunk. Colin McCabe and I have been hard at work the
last 3.5
>> > >> months
>> > >> >> > implementing this feature, and feel that it's reached
a level of
>> > >> >> stability
>> > >> >> > and utility where it's ready for broader testing and
integration.
>> > >> >> >
>> > >> >> > I'd also like to thank Chris Nauroth at Hortonworks for
code
>> > reviews
>> > >> and
>> > >> >> > bug fixes, and everyone who's reviewed the HDFS-4949
design doc
>> and
>> > >> left
>> > >> >> > comments.
>> > >> >> >
>> > >> >> > Obviously, I am +1 for the merge. The vote will run the
standard
>> 7
>> > >> days,
>> > >> >> > closing on October 24 at 11:59PM.
>> > >> >> >
>> > >> >> > Thanks,
>> > >> >> > Andrew
>> > >> >>
>> > >> >
>> > >> > --
>> > >> > CONFIDENTIALITY NOTICE
>> > >> > NOTICE: This message is intended for the use of the individual
or
>> > >> entity to
>> > >> > which it is addressed and may contain information that is
>> > confidential,
>> > >> > privileged and exempt from disclosure under applicable law. If
the
>> > >> reader
>> > >> > of this message is not the intended recipient, you are hereby
>> notified
>> > >> that
>> > >> > any printing, copying, dissemination, distribution, disclosure
or
>> > >> > forwarding of this communication is strictly prohibited. If you
have
>> > >> > received this communication in error, please contact the sender
>> > >> immediately
>> > >> > and delete it from your system. Thank You.
>> > >>
>> > >
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message