hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yongjun Zhang <yzh...@cloudera.com>
Subject Re: Thinking ahead to hadoop-2.6
Date Sun, 19 Oct 2014 18:35:54 GMT
Thanks Arun for the info and reminder.

Hi Alejandro,

HADOOP-10895 is a blocker in the list Arun sent out and it's currently
owned by you. I wonder whether you will have time to work on. If not, would
you mind reassigning it to me and I will spend time on it?

Thanks.

--Yongjun


On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <acm@hortonworks.com> wrote:

> 2.6.0 is close now.
>
> Here are the remaining blockers, I'm hoping cut an RC in the next week or
> so:
> http://s.apache.org/hadoop-2.6.0-blockers
>
> thanks,
> Arun
>
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <acm@hortonworks.com> wrote:
>
> > Folks,
> >
> >  I've created branch-2.6 to stabilize the release.
> >
> >  Committers, please exercise caution henceforth on commits other than
> the ones we've discussed on this thread already.
> >
> >  By default new features should now be targeted to the version "2.7"
> henceforth - I've ensure all the projects have that version on jira.
> >
> > thanks,
> > Arun
> >
> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <acm@hortonworks.com> wrote:
> >
> >> Sounds good. I'll branch this weekend and we can merge the jiras we
> >> discussed in this thread as they they get wrapped next week.
> >>
> >> Thanks everyone.
> >>
> >> Arun
> >>
> >>
> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> We can branch off in a week or two so that work on branch-2 itself can
> go
> >>> ahead with other features that can't fit in 2.6. Independent of that,
> we
> >>> can then decide on the timeline of the release candidates once
> branch-2.6
> >>> is close to being done w.r.t the planned features.
> >>>
> >>> Branching it off can let us focus on specific features that we want in
> for
> >>> 2.6 and then eventually blockers for the release, nothing else. There
> is a
> >>> trivial pain of committing to one more branch, but it's worth it in
> this
> >>> case IMO.
> >>>
> >>> A lot of efforts are happening in parallel from the YARN side from
> where I
> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
> we
> >>> don't branch off and selectively try to get stuff in, it is likely to
> be in
> >>> a perpetual delay.
> >>>
> >>> My 2 cents.
> >>>
> >>> +Vinod
> >>>
> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> >>> wrote:
> >>>
> >>>> Given some of the features are in final stages of stabilization,
> >>>> Arun, we should hold off creating 2.6 branch or building an RC by a
> week?
> >>>> All the features in flux are important ones and worth delaying the
> release
> >>>> by a week.
> >>>>
> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Nicholas,
> >>>>>
> >>>>> My concern about Archival Storage isn't related to the code quality
> or
> >>>> the
> >>>>> size of the feature. I think that you and Jing did good work. My
> concern
> >>>> is
> >>>>> that once we ship, we're locked into that set of archival storage
> APIs,
> >>>> and
> >>>>> these APIs are not yet finalized. Simply being able to turn off
the
> >>>> feature
> >>>>> does not change the compatibility story.
> >>>>>
> >>>>> I'm willing to devote time to help review these JIRAs and kick the
> tires
> >>>> on
> >>>>> the APIs, but my point above was that I'm not sure it'd all be done
> by
> >>>> the
> >>>>> end of the week. Testing might also reveal additional changes that
> need
> >>>> to
> >>>>> be made, which also might not happen by end-of-week.
> >>>>>
> >>>>> I guess the question before us is if we're comfortable putting
> something
> >>>> in
> >>>>> branch-2.6 and then potentially adding API changes after. I'm okay
> with
> >>>>> that as long as we're all aware that this might happen.
> >>>>>
> >>>>> Arun, as RM is this cool with you? Again, I like this feature and
I'm
> >>>> fine
> >>>>> with it's inclusion, just a heads up that we might need some extra
> time
> >>>> to
> >>>>> finalize things before an RC can be cut.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am worry about KMS and transparent encryption since there
are
> quite
> >>>>> many
> >>>>>> bugs discovered after it got merged to branch-2.  It gives us
an
> >>>>> impression
> >>>>>> that the feature is not yet well tested.  Indeed, transparent
> >>>> encryption
> >>>>> is
> >>>>>> a complicated feature which changes the core part of HDFS. 
It is
> not
> >>>>> easy
> >>>>>> to get everything right.
> >>>>>>
> >>>>>>
> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and
low
> risk
> >>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
> of
> >>>>> block
> >>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
> >>>>> storage,
> >>>>>> the blocks will be stored using the appropriate storage types
> specified
> >>>>> by
> >>>>>> storage policies assigned to the files/directories.  Cluster
admin
> >>>> could
> >>>>>> disable the feature by simply not configuring any storage type
and
> not
> >>>>>> setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> >>>> is
> >>>>>> in the final stages to be merged to branch-2.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tsz-Wo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> >>>>>> suresh@hortonworks.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I actually would like to see both archival storage and single
> replica
> >>>>>>> memory writes to be in 2.6 release. Archival storage is
in the
> final
> >>>>>> stages
> >>>>>>> of getting ready for branch-2 merge as Nicholas has already
> indicated
> >>>> on
> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner.
Both
> of
> >>>>> these
> >>>>>>> features are being in development for sometime.
> >>>>>>>
> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> >>>> andrew.wang@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey Arun,
> >>>>>>>>
> >>>>>>>> Maybe we could do a quick run through of the Roadmap
wiki and
> >>>>>> add/retarget
> >>>>>>>> things accordingly?
> >>>>>>>>
> >>>>>>>> I think the KMS and transparent encryption are ready
to go. We've
> >>>> got
> >>>>> a
> >>>>>>>> very few further bug fixes pending, but that's it.
> >>>>>>>>
> >>>>>>>> Two HDFS things that I think probably won't make the
end of the
> week
> >>>>> are
> >>>>>>>> archival storage (HDFS-6584) and single replica memory
writes
> >>>>>> (HDFS-6581),
> >>>>>>>> which I believe are under the HSM banner. HDFS-6484
was just
> merged
> >>>> to
> >>>>>>>> trunk and I think needs a little more work before it
goes into
> >>>>> branch-2.
> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems
a bit
> >>>> further
> >>>>>> off
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Just my 2c as I did not work directly on these features.
I just
> >>>>>> generally
> >>>>>>>> shy away from shipping bits quite this fresh.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Looks like most of the content is in and hadoop-2.6
is shaping up
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> I'll create branch-2.6 by end of the week and we
can go from
> there
> >>>>> to
> >>>>>>>>> stabilize it - hopefully in the next few weeks.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> >>>> acm@hortonworks.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to start
thinking ahead
> >>>> to
> >>>>>>>>>> hadoop-2.6.
> >>>>>>>>>>
> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
> >>>>>>>>>>
> >>>>>>>>>>       • HADOOP
> >>>>>>>>>>               • Credential provider HADOOP-10607
> >>>>>>>>>>       • HDFS
> >>>>>>>>>>               • Heterogeneous storage (Phase
2) - Support APIs
> >>>>> for
> >>>>>>>>> using
> >>>>>>>>>> storage tiers by the applications HDFS-5682
> >>>>>>>>>>               • Memory as storage tier HDFS-5851
> >>>>>>>>>>       • YARN
> >>>>>>>>>>               • Dynamic Resource Configuration
YARN-291
> >>>>>>>>>>               • NodeManager Restart YARN-1336
> >>>>>>>>>>               • ResourceManager HA Phase 2
YARN-556
> >>>>>>>>>>               • Support for admin-specified
labels in YARN
> >>>>>> YARN-796
> >>>>>>>>>>               • Support for automatic, shared
cache for YARN
> >>>>>>>>> application
> >>>>>>>>>> artifacts YARN-1492
> >>>>>>>>>>               • Support NodeGroup layer topology
on YARN
> >>>> YARN-18
> >>>>>>>>>>               • Support for Docker containers
in YARN
> >>>> YARN-1964
> >>>>>>>>>>               • YARN service registry YARN-913
> >>>>>>>>>>
> >>>>>>>>>> My suspicion is, as is normal, some will make
the cut and some
> >>>>>> won't.
> >>>>>>>>>> Please do add/subtract from the list as appropriate.
Ideally, it
> >>>>>> would
> >>>>>>>> be
> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say,
October) to keep
> >>>> up a
> >>>>>>>>> cadence.
> >>>>>>>>>>
> >>>>>>>>>> More importantly, as we discussed previously,
we'd like
> >>>>> hadoop-2.6
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the *last* Apache Hadoop 2.x release which support
JDK6. I'll
> >>>>> start
> >>>>>> a
> >>>>>>>>>> discussion with other communities (HBase, Pig,
Hive, Oozie etc.)
> >>>>> and
> >>>>>>>> see
> >>>>>>>>>> how they feel about this.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of
the individual or
> >>>>>> entity
> >>>>>>>> to
> >>>>>>>>> which it is addressed and may contain information
that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable
law. If
> the
> >>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you
are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution,
disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited.
If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact
the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual
or
> >>>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law.
If the
> >>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure
or
> >>>>>>> forwarding of this communication is strictly prohibited.
If you
> have
> >>>>>>> received this communication in error, please contact the
sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> immediately
> >>> and delete it from your system. Thank You.
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/hdp/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message