hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yongjun Zhang <yzh...@cloudera.com>
Subject Re: Thinking ahead to hadoop-2.6
Date Fri, 24 Oct 2014 05:44:43 GMT
Hi,

I posted a patch to hadoop-10895 couple of days back. Wonder if anyone
could help doing a review?

Thanks a lot.

--Yongjun


On Sun, Oct 19, 2014 at 11:35 AM, Yongjun Zhang <yzhang@cloudera.com> wrote:

> Thanks Arun for the info and reminder.
>
> Hi Alejandro,
>
> HADOOP-10895 is a blocker in the list Arun sent out and it's currently
> owned by you. I wonder whether you will have time to work on. If not, would
> you mind reassigning it to me and I will spend time on it?
>
> Thanks.
>
> --Yongjun
>
>
> On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <acm@hortonworks.com>
> wrote:
>
>> 2.6.0 is close now.
>>
>> Here are the remaining blockers, I'm hoping cut an RC in the next week or
>> so:
>> http://s.apache.org/hadoop-2.6.0-blockers
>>
>> thanks,
>> Arun
>>
>> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <acm@hortonworks.com> wrote:
>>
>> > Folks,
>> >
>> >  I've created branch-2.6 to stabilize the release.
>> >
>> >  Committers, please exercise caution henceforth on commits other than
>> the ones we've discussed on this thread already.
>> >
>> >  By default new features should now be targeted to the version "2.7"
>> henceforth - I've ensure all the projects have that version on jira.
>> >
>> > thanks,
>> > Arun
>> >
>> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <acm@hortonworks.com> wrote:
>> >
>> >> Sounds good. I'll branch this weekend and we can merge the jiras we
>> >> discussed in this thread as they they get wrapped next week.
>> >>
>> >> Thanks everyone.
>> >>
>> >> Arun
>> >>
>> >>
>> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
>> vinodkv@apache.org> wrote:
>> >>>
>> >>> We can branch off in a week or two so that work on branch-2 itself
>> can go
>> >>> ahead with other features that can't fit in 2.6. Independent of that,
>> we
>> >>> can then decide on the timeline of the release candidates once
>> branch-2.6
>> >>> is close to being done w.r.t the planned features.
>> >>>
>> >>> Branching it off can let us focus on specific features that we want
>> in for
>> >>> 2.6 and then eventually blockers for the release, nothing else. There
>> is a
>> >>> trivial pain of committing to one more branch, but it's worth it in
>> this
>> >>> case IMO.
>> >>>
>> >>> A lot of efforts are happening in parallel from the YARN side from
>> where I
>> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
>> we
>> >>> don't branch off and selectively try to get stuff in, it is likely to
>> be in
>> >>> a perpetual delay.
>> >>>
>> >>> My 2 cents.
>> >>>
>> >>> +Vinod
>> >>>
>> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
>> suresh@hortonworks.com>
>> >>> wrote:
>> >>>
>> >>>> Given some of the features are in final stages of stabilization,
>> >>>> Arun, we should hold off creating 2.6 branch or building an RC by
a
>> week?
>> >>>> All the features in flux are important ones and worth delaying the
>> release
>> >>>> by a week.
>> >>>>
>> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
>> andrew.wang@cloudera.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Hey Nicholas,
>> >>>>>
>> >>>>> My concern about Archival Storage isn't related to the code
quality
>> or
>> >>>> the
>> >>>>> size of the feature. I think that you and Jing did good work.
My
>> concern
>> >>>> is
>> >>>>> that once we ship, we're locked into that set of archival storage
>> APIs,
>> >>>> and
>> >>>>> these APIs are not yet finalized. Simply being able to turn
off the
>> >>>> feature
>> >>>>> does not change the compatibility story.
>> >>>>>
>> >>>>> I'm willing to devote time to help review these JIRAs and kick
the
>> tires
>> >>>> on
>> >>>>> the APIs, but my point above was that I'm not sure it'd all
be done
>> by
>> >>>> the
>> >>>>> end of the week. Testing might also reveal additional changes
that
>> need
>> >>>> to
>> >>>>> be made, which also might not happen by end-of-week.
>> >>>>>
>> >>>>> I guess the question before us is if we're comfortable putting
>> something
>> >>>> in
>> >>>>> branch-2.6 and then potentially adding API changes after. I'm
okay
>> with
>> >>>>> that as long as we're all aware that this might happen.
>> >>>>>
>> >>>>> Arun, as RM is this cool with you? Again, I like this feature
and
>> I'm
>> >>>> fine
>> >>>>> with it's inclusion, just a heads up that we might need some
extra
>> time
>> >>>> to
>> >>>>> finalize things before an RC can be cut.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Andrew
>> >>>>>
>> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>> >>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> I am worry about KMS and transparent encryption since there
are
>> quite
>> >>>>> many
>> >>>>>> bugs discovered after it got merged to branch-2.  It gives
us an
>> >>>>> impression
>> >>>>>> that the feature is not yet well tested.  Indeed, transparent
>> >>>> encryption
>> >>>>> is
>> >>>>>> a complicated feature which changes the core part of HDFS.
 It is
>> not
>> >>>>> easy
>> >>>>>> to get everything right.
>> >>>>>>
>> >>>>>>
>> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple
and low
>> risk
>> >>>>>> feature.  It introduces a new storage type ARCHIVE and the
concept
>> of
>> >>>>> block
>> >>>>>> storage policy to HDFS.  When a cluster is configured with
ARCHIVE
>> >>>>> storage,
>> >>>>>> the blocks will be stored using the appropriate storage
types
>> specified
>> >>>>> by
>> >>>>>> storage policies assigned to the files/directories.  Cluster
admin
>> >>>> could
>> >>>>>> disable the feature by simply not configuring any storage
type and
>> not
>> >>>>>> setting any storage policy as before.   As Suresh mentioned,
>> HDFS-6584
>> >>>> is
>> >>>>>> in the final stages to be merged to branch-2.
>> >>>>>>
>> >>>>>> Regards,
>> >>>>>> Tsz-Wo
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas
<
>> >>>>>> suresh@hortonworks.com> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> I actually would like to see both archival storage and
single
>> replica
>> >>>>>>> memory writes to be in 2.6 release. Archival storage
is in the
>> final
>> >>>>>> stages
>> >>>>>>> of getting ready for branch-2 merge as Nicholas has
already
>> indicated
>> >>>> on
>> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready
sooner. Both
>> of
>> >>>>> these
>> >>>>>>> features are being in development for sometime.
>> >>>>>>>
>> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>> >>>> andrew.wang@cloudera.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hey Arun,
>> >>>>>>>>
>> >>>>>>>> Maybe we could do a quick run through of the Roadmap
wiki and
>> >>>>>> add/retarget
>> >>>>>>>> things accordingly?
>> >>>>>>>>
>> >>>>>>>> I think the KMS and transparent encryption are ready
to go. We've
>> >>>> got
>> >>>>> a
>> >>>>>>>> very few further bug fixes pending, but that's it.
>> >>>>>>>>
>> >>>>>>>> Two HDFS things that I think probably won't make
the end of the
>> week
>> >>>>> are
>> >>>>>>>> archival storage (HDFS-6584) and single replica
memory writes
>> >>>>>> (HDFS-6581),
>> >>>>>>>> which I believe are under the HSM banner. HDFS-6484
was just
>> merged
>> >>>> to
>> >>>>>>>> trunk and I think needs a little more work before
it goes into
>> >>>>> branch-2.
>> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet,
so seems a bit
>> >>>> further
>> >>>>>> off
>> >>>>>>>> yet.
>> >>>>>>>>
>> >>>>>>>> Just my 2c as I did not work directly on these features.
I just
>> >>>>>> generally
>> >>>>>>>> shy away from shipping bits quite this fresh.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Andrew
>> >>>>>>>>
>> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <
>> acm@hortonworks.com>
>> >>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Looks like most of the content is in and hadoop-2.6
is shaping
>> up
>> >>>>>> nicely.
>> >>>>>>>>>
>> >>>>>>>>> I'll create branch-2.6 by end of the week and
we can go from
>> there
>> >>>>> to
>> >>>>>>>>> stabilize it - hopefully in the next few weeks.
>> >>>>>>>>>
>> >>>>>>>>> Thoughts?
>> >>>>>>>>>
>> >>>>>>>>> thanks,
>> >>>>>>>>> Arun
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy
<
>> >>>> acm@hortonworks.com
>> >>>>>>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Folks,
>> >>>>>>>>>>
>> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to
start thinking ahead
>> >>>> to
>> >>>>>>>>>> hadoop-2.6.
>> >>>>>>>>>>
>> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
>> >>>>>>>>>>
>> >>>>>>>>>>       • HADOOP
>> >>>>>>>>>>               • Credential provider HADOOP-10607
>> >>>>>>>>>>       • HDFS
>> >>>>>>>>>>               • Heterogeneous storage
(Phase 2) - Support APIs
>> >>>>> for
>> >>>>>>>>> using
>> >>>>>>>>>> storage tiers by the applications HDFS-5682
>> >>>>>>>>>>               • Memory as storage tier
HDFS-5851
>> >>>>>>>>>>       • YARN
>> >>>>>>>>>>               • Dynamic Resource Configuration
YARN-291
>> >>>>>>>>>>               • NodeManager Restart YARN-1336
>> >>>>>>>>>>               • ResourceManager HA Phase
2 YARN-556
>> >>>>>>>>>>               • Support for admin-specified
labels in YARN
>> >>>>>> YARN-796
>> >>>>>>>>>>               • Support for automatic,
shared cache for YARN
>> >>>>>>>>> application
>> >>>>>>>>>> artifacts YARN-1492
>> >>>>>>>>>>               • Support NodeGroup layer
topology on YARN
>> >>>> YARN-18
>> >>>>>>>>>>               • Support for Docker containers
in YARN
>> >>>> YARN-1964
>> >>>>>>>>>>               • YARN service registry
YARN-913
>> >>>>>>>>>>
>> >>>>>>>>>> My suspicion is, as is normal, some will
make the cut and some
>> >>>>>> won't.
>> >>>>>>>>>> Please do add/subtract from the list as
appropriate. Ideally,
>> it
>> >>>>>> would
>> >>>>>>>> be
>> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say,
October) to keep
>> >>>> up a
>> >>>>>>>>> cadence.
>> >>>>>>>>>>
>> >>>>>>>>>> More importantly, as we discussed previously,
we'd like
>> >>>>> hadoop-2.6
>> >>>>>> to
>> >>>>>>>> be
>> >>>>>>>>>> the *last* Apache Hadoop 2.x release which
support JDK6. I'll
>> >>>>> start
>> >>>>>> a
>> >>>>>>>>>> discussion with other communities (HBase,
Pig, Hive, Oozie
>> etc.)
>> >>>>> and
>> >>>>>>>> see
>> >>>>>>>>>> how they feel about this.
>> >>>>>>>>>>
>> >>>>>>>>>> thanks,
>> >>>>>>>>>> Arun
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> Arun C. Murthy
>> >>>>>>>>> Hortonworks Inc.
>> >>>>>>>>> http://hortonworks.com/
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> CONFIDENTIALITY NOTICE
>> >>>>>>>>> NOTICE: This message is intended for the use
of the individual
>> or
>> >>>>>> entity
>> >>>>>>>> to
>> >>>>>>>>> which it is addressed and may contain information
that is
>> >>>>>> confidential,
>> >>>>>>>>> privileged and exempt from disclosure under
applicable law. If
>> the
>> >>>>>> reader
>> >>>>>>>>> of this message is not the intended recipient,
you are hereby
>> >>>>> notified
>> >>>>>>>> that
>> >>>>>>>>> any printing, copying, dissemination, distribution,
disclosure
>> or
>> >>>>>>>>> forwarding of this communication is strictly
prohibited. If you
>> >>>> have
>> >>>>>>>>> received this communication in error, please
contact the sender
>> >>>>>>>> immediately
>> >>>>>>>>> and delete it from your system. Thank You.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> http://hortonworks.com/download/
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> CONFIDENTIALITY NOTICE
>> >>>>>>> NOTICE: This message is intended for the use of the
individual or
>> >>>> entity
>> >>>>>> to
>> >>>>>>> which it is addressed and may contain information that
is
>> >>>> confidential,
>> >>>>>>> privileged and exempt from disclosure under applicable
law. If the
>> >>>>> reader
>> >>>>>>> of this message is not the intended recipient, you are
hereby
>> notified
>> >>>>>> that
>> >>>>>>> any printing, copying, dissemination, distribution,
disclosure or
>> >>>>>>> forwarding of this communication is strictly prohibited.
If you
>> have
>> >>>>>>> received this communication in error, please contact
the sender
>> >>>>>> immediately
>> >>>>>>> and delete it from your system. Thank You.
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> http://hortonworks.com/download/
>> >>>>
>> >>>> --
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >>>> which it is addressed and may contain information that is
>> confidential,
>> >>>> privileged and exempt from disclosure under applicable law. If the
>> reader
>> >>>> of this message is not the intended recipient, you are hereby
>> notified that
>> >>>> any printing, copying, dissemination, distribution, disclosure or
>> >>>> forwarding of this communication is strictly prohibited. If you
have
>> >>>> received this communication in error, please contact the sender
>> immediately
>> >>>> and delete it from your system. Thank You.
>> >>>
>> >>> --
>> >>> CONFIDENTIALITY NOTICE
>> >>> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >>> which it is addressed and may contain information that is
>> confidential,
>> >>> privileged and exempt from disclosure under applicable law. If the
>> reader
>> >>> of this message is not the intended recipient, you are hereby
>> notified that
>> >>> any printing, copying, dissemination, distribution, disclosure or
>> >>> forwarding of this communication is strictly prohibited. If you have
>> >>> received this communication in error, please contact the sender
>> immediately
>> >>> and delete it from your system. Thank You.
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/hdp/
>> >
>> >
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/hdp/
>>
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message