hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Thinking ahead to hadoop-2.6
Date Wed, 05 Nov 2014 13:26:44 GMT
Looks like most of the big ones are in. I'll try prod people into finishing up the rest and
get an RC going this week.

Thanks everyone!

Arun

On Oct 14, 2014, at 11:25 AM, Arun C Murthy <acm@hortonworks.com> wrote:

> 2.6.0 is close now.
> 
> Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
> http://s.apache.org/hadoop-2.6.0-blockers
> 
> thanks,
> Arun
> 
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <acm@hortonworks.com> wrote:
> 
>> Folks,
>> 
>>  I've created branch-2.6 to stabilize the release.
>>  
>>  Committers, please exercise caution henceforth on commits other than the ones we've
discussed on this thread already.
>> 
>>  By default new features should now be targeted to the version "2.7" henceforth -
I've ensure all the projects have that version on jira.
>> 
>> thanks,
>> Arun
>> 
>> On Sep 26, 2014, at 1:08 AM, Arun Murthy <acm@hortonworks.com> wrote:
>> 
>>> Sounds good. I'll branch this weekend and we can merge the jiras we
>>> discussed in this thread as they they get wrapped next week.
>>> 
>>> Thanks everyone.
>>> 
>>> Arun
>>> 
>>> 
>>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org>
wrote:
>>>> 
>>>> We can branch off in a week or two so that work on branch-2 itself can go
>>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>>> can then decide on the timeline of the release candidates once branch-2.6
>>>> is close to being done w.r.t the planned features.
>>>> 
>>>> Branching it off can let us focus on specific features that we want in for
>>>> 2.6 and then eventually blockers for the release, nothing else. There is
a
>>>> trivial pain of committing to one more branch, but it's worth it in this
>>>> case IMO.
>>>> 
>>>> A lot of efforts are happening in parallel from the YARN side from where
I
>>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>>> don't branch off and selectively try to get stuff in, it is likely to be
in
>>>> a perpetual delay.
>>>> 
>>>> My 2 cents.
>>>> 
>>>> +Vinod
>>>> 
>>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <suresh@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> Given some of the features are in final stages of stabilization,
>>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>>> All the features in flux are important ones and worth delaying the release
>>>>> by a week.
>>>>> 
>>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <andrew.wang@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Nicholas,
>>>>>> 
>>>>>> My concern about Archival Storage isn't related to the code quality
or
>>>>> the
>>>>>> size of the feature. I think that you and Jing did good work. My
concern
>>>>> is
>>>>>> that once we ship, we're locked into that set of archival storage
APIs,
>>>>> and
>>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>>> feature
>>>>>> does not change the compatibility story.
>>>>>> 
>>>>>> I'm willing to devote time to help review these JIRAs and kick the
tires
>>>>> on
>>>>>> the APIs, but my point above was that I'm not sure it'd all be done
by
>>>>> the
>>>>>> end of the week. Testing might also reveal additional changes that
need
>>>>> to
>>>>>> be made, which also might not happen by end-of-week.
>>>>>> 
>>>>>> I guess the question before us is if we're comfortable putting something
>>>>> in
>>>>>> branch-2.6 and then potentially adding API changes after. I'm okay
with
>>>>>> that as long as we're all aware that this might happen.
>>>>>> 
>>>>>> Arun, as RM is this cool with you? Again, I like this feature and
I'm
>>>>> fine
>>>>>> with it's inclusion, just a heads up that we might need some extra
time
>>>>> to
>>>>>> finalize things before an RC can be cut.
>>>>>> 
>>>>>> Thanks,
>>>>>> Andrew
>>>>>> 
>>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am worry about KMS and transparent encryption since there are
quite
>>>>>> many
>>>>>>> bugs discovered after it got merged to branch-2.  It gives us
an
>>>>>> impression
>>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>>> encryption
>>>>>> is
>>>>>>> a complicated feature which changes the core part of HDFS.  It
is not
>>>>>> easy
>>>>>>> to get everything right.
>>>>>>> 
>>>>>>> 
>>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and
low risk
>>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
of
>>>>>> block
>>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>>> storage,
>>>>>>> the blocks will be stored using the appropriate storage types
specified
>>>>>> by
>>>>>>> storage policies assigned to the files/directories.  Cluster
admin
>>>>> could
>>>>>>> disable the feature by simply not configuring any storage type
and not
>>>>>>> setting any storage policy as before.   As Suresh mentioned,
HDFS-6584
>>>>> is
>>>>>>> in the final stages to be merged to branch-2.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Tsz-Wo
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>>> suresh@hortonworks.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I actually would like to see both archival storage and single
replica
>>>>>>>> memory writes to be in 2.6 release. Archival storage is in
the final
>>>>>>> stages
>>>>>>>> of getting ready for branch-2 merge as Nicholas has already
indicated
>>>>> on
>>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner.
Both of
>>>>>> these
>>>>>>>> features are being in development for sometime.
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>>> andrew.wang@cloudera.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Arun,
>>>>>>>>> 
>>>>>>>>> Maybe we could do a quick run through of the Roadmap
wiki and
>>>>>>> add/retarget
>>>>>>>>> things accordingly?
>>>>>>>>> 
>>>>>>>>> I think the KMS and transparent encryption are ready
to go. We've
>>>>> got
>>>>>> a
>>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>>> 
>>>>>>>>> Two HDFS things that I think probably won't make the
end of the week
>>>>>> are
>>>>>>>>> archival storage (HDFS-6584) and single replica memory
writes
>>>>>>> (HDFS-6581),
>>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was
just merged
>>>>> to
>>>>>>>>> trunk and I think needs a little more work before it
goes into
>>>>>> branch-2.
>>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems
a bit
>>>>> further
>>>>>>> off
>>>>>>>>> yet.
>>>>>>>>> 
>>>>>>>>> Just my 2c as I did not work directly on these features.
I just
>>>>>>> generally
>>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Looks like most of the content is in and hadoop-2.6
is shaping up
>>>>>>> nicely.
>>>>>>>>>> 
>>>>>>>>>> I'll create branch-2.6 by end of the week and we
can go from there
>>>>>> to
>>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>>> acm@hortonworks.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Folks,
>>>>>>>>>>> 
>>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start
thinking ahead
>>>>> to
>>>>>>>>>>> hadoop-2.6.
>>>>>>>>>>> 
>>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>>> 
>>>>>>>>>>>       • HADOOP
>>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>>       • HDFS
>>>>>>>>>>>               • Heterogeneous storage (Phase
2) - Support APIs
>>>>>> for
>>>>>>>>>> using
>>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>>       • YARN
>>>>>>>>>>>               • Dynamic Resource Configuration
YARN-291
>>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>>               • ResourceManager HA Phase 2
YARN-556
>>>>>>>>>>>               • Support for admin-specified
labels in YARN
>>>>>>> YARN-796
>>>>>>>>>>>               • Support for automatic, shared
cache for YARN
>>>>>>>>>> application
>>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>>               • Support NodeGroup layer topology
on YARN
>>>>> YARN-18
>>>>>>>>>>>               • Support for Docker containers
in YARN
>>>>> YARN-1964
>>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>>> 
>>>>>>>>>>> My suspicion is, as is normal, some will make
the cut and some
>>>>>>> won't.
>>>>>>>>>>> Please do add/subtract from the list as appropriate.
Ideally, it
>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say,
October) to keep
>>>>> up a
>>>>>>>>>> cadence.
>>>>>>>>>>> 
>>>>>>>>>>> More importantly, as we discussed previously,
we'd like
>>>>>> hadoop-2.6
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>> the *last* Apache Hadoop 2.x release which support
JDK6. I'll
>>>>>> start
>>>>>>> a
>>>>>>>>>>> discussion with other communities (HBase, Pig,
Hive, Oozie etc.)
>>>>>> and
>>>>>>>>> see
>>>>>>>>>>> how they feel about this.
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Arun C. Murthy
>>>>>>>>>> Hortonworks Inc.
>>>>>>>>>> http://hortonworks.com/
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the
individual or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information
that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable
law. If the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you
are hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution,
disclosure or
>>>>>>>>>> forwarding of this communication is strictly prohibited.
If you
>>>>> have
>>>>>>>>>> received this communication in error, please contact
the sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> http://hortonworks.com/download/
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual
or
>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law.
If the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby
notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure
or
>>>>>>>> forwarding of this communication is strictly prohibited.
If you have
>>>>>>>> received this communication in error, please contact the
sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity
to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified
that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity
to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/hdp/
>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message