hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo \(Nicholas\), Sze" <s29752-hadoop...@yahoo.com.INVALID>
Subject Re: Thinking ahead to hadoop-2.6
Date Wed, 24 Sep 2014 02:30:15 GMT
Hi,

I am worry about KMS and transparent encryption since there are quite many bugs discovered
after it got merged to branch-2.  It gives us an impression that the feature is not yet well
tested.  Indeed, transparent encryption is a complicated feature which changes the core part
of HDFS.  It is not easy to get everything right.


For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature.  It introduces
a new storage type ARCHIVE and the concept of block storage policy to HDFS.  When a cluster
is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage
types specified by storage policies assigned to the files/directories.  Cluster admin could
disable the feature by simply not configuring any storage type and not setting any storage
policy as before.   As Suresh mentioned, HDFS-6584 is in the final stages to be merged to
branch-2.

Regards,
Tsz-Wo



On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <suresh@hortonworks.com> wrote:
 

>
>
>I actually would like to see both archival storage and single replica
>memory writes to be in 2.6 release. Archival storage is in the final stages
>of getting ready for branch-2 merge as Nicholas has already indicated on
>the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
>features are being in development for sometime.
>
>On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <andrew.wang@cloudera.com>
>wrote:
>
>> Hey Arun,
>>
>> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
>> things accordingly?
>>
>> I think the KMS and transparent encryption are ready to go. We've got a
>> very few further bug fixes pending, but that's it.
>>
>> Two HDFS things that I think probably won't make the end of the week are
>> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
>> which I believe are under the HSM banner. HDFS-6484 was just merged to
>> trunk and I think needs a little more work before it goes into branch-2.
>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
>> yet.
>>
>> Just my 2c as I did not work directly on these features. I just generally
>> shy away from shipping bits quite this fresh.
>>
>> Thanks,
>> Andrew
>>
>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com> wrote:
>>
>> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>> >
>> > I'll create branch-2.6 by end of the week and we can go from there to
>> > stabilize it - hopefully in the next few weeks.
>> >
>> > Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <acm@hortonworks.com>
>> > wrote:
>> >
>> > > Folks,
>> > >
>> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> > > hadoop-2.6.
>> > >
>> > >  Currently, here is the Roadmap per the wiki:
>> > >
>> > >         • HADOOP
>> > >                 • Credential provider HADOOP-10607
>> > >         • HDFS
>> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
>> > using
>> > > storage tiers by the applications HDFS-5682
>> > >                 • Memory as storage tier HDFS-5851
>> > >         • YARN
>> > >                 • Dynamic Resource Configuration YARN-291
>> > >                 • NodeManager Restart YARN-1336
>> > >                 • ResourceManager HA Phase 2 YARN-556
>> > >                 • Support for admin-specified labels in YARN YARN-796
>> > >                 • Support for automatic, shared cache for YARN
>> > application
>> > > artifacts YARN-1492
>> > >                 • Support NodeGroup layer topology on YARN YARN-18
>> > >                 • Support for Docker containers in YARN YARN-1964
>> > >                 • YARN service registry YARN-913
>> > >
>> > >  My suspicion is, as is normal, some will make the cut and some won't.
>> > > Please do add/subtract from the list as appropriate. Ideally, it would
>> be
>> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
>> > cadence.
>> > >
>> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
>> be
>> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
>> see
>> > > how they feel about this.
>> > >
>> > > thanks,
>> > > Arun
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
>
>-- 
>http://hortonworks.com/download/
>
>-- 
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity to 
>which it is addressed and may contain information that is confidential, 
>privileged and exempt from disclosure under applicable law. If the reader 
>of this message is not the intended recipient, you are hereby notified that 
>any printing, copying, dissemination, distribution, disclosure or 
>forwarding of this communication is strictly prohibited. If you have 
>received this communication in error, please contact the sender immediately 
>and delete it from your system. Thank You.
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message