hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: [Discussion] Merge YARN-321 into Branch-2
Date Tue, 07 Jan 2014 13:52:05 GMT
Zhijie,

Sounds good.

On the last part of my previous email, lets wait till  YARN-321 is updated
to latest trunk.

> I would like to see #1 and #2 taken care before making a decision.
> The reason for this is that if the source changes outside of the AHS
> are too pervasive, then we may end up be making difficult backports
> from trunk to branch-2 and release branches because of the delta.
>
> Can we get a rebase of the YARN-321 to the head of trunk to see if
> my concerns are valid or not?

Thanks.


On Mon, Jan 6, 2014 at 8:21 PM, Zhijie Shen <zshen@hortonworks.com> wrote:

> Hi folks,
>
> Thanks for replying. Please see the response bellow.
>
> bq. 2* The last update of YARN-321 was done in NOV10, this was done
> from branch-2
> (that seems a NIT as it should be against trunk)
>
> Basically it's a discussion thread. I'm already in the process of updating
> the branch. As I mentioned before, one of the motivation of merge branch
> YARN-321 is to prevent it from further falling behind.
>
> bq. 1* The merge must be done in trunk first (then, ideally, from trunk
> into branch-2)
> bq. 3* IMO, until we don’t have security, we should merge into trunk only
>
> Yes, we plan to merge to trunk, then to branch-2, but I agree, let's have
> security ready before going to branch-2. We can continue with security on
> trunk.
>
> bq. Regarding doc, while we don't necessarily need full documentation
> before merging, my feeling is that we should at least have a page on it
> that will allow give cluster operators a sketch of its role, APIs, and
> implications.
>
> Agree, let's prepare a doc as well.
>
> bq. My opinion is that we should mark each API @Stable unless there is a
> strong reason for it not to be.  When I say APIs, I am thinking of the REST
> APIs, RPC interface, and RM<->AHS shared-bus.
>
> I'm afraid it's still too early to confirm which APIs can be marked
> @Stable. How about not doing this until security is ready and refactoring
> duplicate code is done? At that time, we should have a clearer picture
> about it.
>
> Thanks,
> Zhijie
>
>
> On Mon, Jan 6, 2014 at 3:02 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > This is great news. A few things to consider before doing the merge:
> >
> > 1* The merge must be done in trunk first (then, ideally, from trunk into
> > branch-2)
> > 2* The last update of YARN-321 was done in NOV10, this was done from
> > branch-2 (that seems a NIT as it should be against trunk)
> > 3* IMO, until we don’t have security, we should merge into trunk only
> >
> > I would like to see #1 and #2 taken care before making a decision. The
> > reason for this is that if the source changes outside of the AHS are too
> > pervasive, then we may end up be making difficult backports from trunk to
> > branch-2 and release branches because of the delta.
> >
> > Can we get a rebase of the YARN-321 to the head of trunk to see if my
> > concerns are valid or not?
> >
> > Thanks.
> >
> > Alejandro
> >
> >
> >
> > On Mon, Jan 6, 2014 at 1:11 AM, Sandy Ryza <sandy.ryza@cloudera.com>
> > wrote:
> >
> > > Very excited for this feature and appreciative of all the work put into
> > > this.  Reviewed the JIRA and my only two remaining concerns are about
> > > documentation and API stability.
> > >
> > > Regarding doc, while we don't necessarily need full documentation
> before
> > > merging, my feeling is that we should at least have a page on it that
> > will
> > > allow give cluster operators a sketch of its role, APIs, and
> > implications.
> > >  If this doesn't exist yet, would be happy to help with reviewing.
> > >
> > > Regarding APIs, I imagine some will jump on this as soon as it becomes
> > part
> > > of a release as it's a fairly essential feature for non-MR apps.  My
> > > opinion is that we should mark each API @Stable unless there is a
> strong
> > > reason for it not to be.  When I say APIs, I am thinking of the REST
> > APIs,
> > > RPC interface, and RM<->AHS shared-bus.
> > >
> > > -Sandy
> > >
> > >
> > > On Sun, Jan 5, 2014 at 8:33 PM, lohit <lohit.vijayarenu@gmail.com>
> > wrote:
> > >
> > > > +1 to merge into branch2 now.
> > > > On Jan 5, 2014 6:22 PM, "Zhijie Shen" <zshen@hortonworks.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Majority of the functionality of Application History Server has
> been
> > > > > completed on branch YARN-321. AHS can now work end-to-end.
> > > > ResourceManager
> > > > > records the historical information of the application, the
> > application
> > > > > attempt and the container in terms of events via a history writer
> on
> > a
> > > > > separate thread. The historical information is going to be
> persisted
> > on
> > > > > HDFS. On the other side, an application history server runs as a
> > > separate
> > > > > process, and it allows users to access the historical information
> via
> > > RPC
> > > > > interface, web UI and REST APIs.
> > > > >
> > > > > According to the proposal, the only major missing piece is
> security.
> > > > Except
> > > > > it, YARN-321 should be pretty much ready to be merged to Branch-2.
> We
> > > > > propose to merge YARN-321 to Branch-2 now, such that we can prevent
> > > > > YARN-321 from falling behind too much, and reduce the effort of
> > editing
> > > > > merge conflicts. After merge, we can continue on security,
> > refactoring
> > > > > duplicate code, fixing bugs and other improvements.
> > > > >
> > > > > Anyone who is interested in looking at AHS can review/play with
> > > YARN-321
> > > > > branch:
> > http://svn.apache.org/viewvc/hadoop/common/branches/YARN-321/
> > > > >
> > > > > You can also have a look at the latest design doc:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12619638/Generic%20Application%20History%20-%20Design-20131219.pdf
> > > > >
> > > > > If there are no objections, we'll push towards updating the branch,
> > > > running
> > > > > the patch through Jenkins and getting ready for merge vote by the
> end
> > > of
> > > > > this week.
> > > > >
> > > > > Thanks,
> > > > > Zhijie
> > > > >
> > > > > --
> > > > > Zhijie Shen
> > > > > Hortonworks Inc.
> > > > > http://hortonworks.com/
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Alejandro

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message