From hdfs-dev-return-15794-apmail-hadoop-hdfs-dev-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 24 02:30:47 2014 Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0135B110A1 for ; Wed, 24 Sep 2014 02:30:47 +0000 (UTC) Received: (qmail 28673 invoked by uid 500); 24 Sep 2014 02:30:45 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 28567 invoked by uid 500); 24 Sep 2014 02:30:45 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 28107 invoked by uid 99); 24 Sep 2014 02:30:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Sep 2014 02:30:45 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,REPTO_QUOTE_YAHOO,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of s29752-hadoopdev@yahoo.com designates 98.138.91.140 as permitted sender) Received: from [98.138.91.140] (HELO nm10-vm3.bullet.mail.ne1.yahoo.com) (98.138.91.140) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Sep 2014 02:30:17 +0000 Received: from [98.138.101.132] by nm10.bullet.mail.ne1.yahoo.com with NNFMP; 24 Sep 2014 02:30:15 -0000 Received: from [98.138.87.12] by tm20.bullet.mail.ne1.yahoo.com with NNFMP; 24 Sep 2014 02:30:15 -0000 Received: from [127.0.0.1] by omp1012.mail.ne1.yahoo.com with NNFMP; 24 Sep 2014 02:30:15 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 871366.46237.bm@omp1012.mail.ne1.yahoo.com Received: (qmail 77233 invoked by uid 60001); 24 Sep 2014 02:30:15 -0000 X-YMail-OSG: vIsbog8VM1lGpBBJBsGu5tksP_z1Vz0rzDyFgvl20ED4mPL n3Wd1ko.rAxY4_anwfFcqAXCtI02slMLToXvB4zfm9W0Q_fM.2C_5SDzSGV3 vfxBXOtc_X4q596eI7il_MscdgOVSTZxJ6_0tzQfSoEVosGZpwlxAe8bq_op y1B2tNGt0S_0J9eGPCaN56NwusID59vLgAiYW_4QDH0yl9zQt1i.rWe2Kmx4 4Z9ybTM5zrTX.Zwb7hky9zPD2sw.0oP85MepzozchBBlF279Jsn5WR4XWUpc ocCoiebvvJ4v2vqIz8l39y6eJCsQ_bwZHwiQ6SFCRNKeK9OjCLz7SVYEqCE8 TbtCVB3GlZpkEMZwNfrM5PTr7BYojWu01pk36NtjhApNyCW79clLmkY0InQf agF2J3Lf6qaardI1MyIkReGm5JAMboON.RmLuQF2HZd8hv0nETzjeY6V0DIa gVViHr92i6ZPj3nZM8c3enufOtOzhz7TQ1OcSb9o8dxcbbsP5SOulyDLrs0v u9IxgDeKZ6qP181C9aWEWzdO8TwMYZZpdrOalDz87HMkiaGCs02cTvnHTOAM TBzQbo5XXRPqb8SmSIeZlm2EC0rm1e8Y.qN7KVxm92ZDz87XHeeAicDtR3f1 r Received: from [113.252.95.250] by web120004.mail.ne1.yahoo.com via HTTP; Tue, 23 Sep 2014 19:30:15 PDT X-Rocket-MIMEInfo: 002.001,SGksCgpJIGFtIHdvcnJ5IGFib3V0IEtNUyBhbmQgdHJhbnNwYXJlbnQgZW5jcnlwdGlvbiBzaW5jZSB0aGVyZSBhcmUgcXVpdGUgbWFueSBidWdzIGRpc2NvdmVyZWQgYWZ0ZXIgaXQgZ290IG1lcmdlZCB0byBicmFuY2gtMi4gIEl0IGdpdmVzIHVzIGFuIGltcHJlc3Npb24gdGhhdCB0aGUgZmVhdHVyZSBpcyBub3QgeWV0IHdlbGwgdGVzdGVkLiAgSW5kZWVkLCB0cmFuc3BhcmVudCBlbmNyeXB0aW9uIGlzIGEgY29tcGxpY2F0ZWQgZmVhdHVyZSB3aGljaCBjaGFuZ2VzIHRoZSBjb3JlIHBhcnQgb2YgSERGUy4BMAEBAQE- X-Mailer: YahooMailWebService/0.8.203.696 References: Message-ID: <1411525815.61307.YahooMailNeo@web120004.mail.ne1.yahoo.com> Date: Tue, 23 Sep 2014 19:30:15 -0700 From: "Tsz Wo \(Nicholas\), Sze" Reply-To: "Tsz Wo \(Nicholas\), Sze" Subject: Re: Thinking ahead to hadoop-2.6 To: "hdfs-dev@hadoop.apache.org" , "common-dev@hadoop.apache.org" Cc: "mapreduce-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="1265908415-1107087233-1411525815=:61307" X-Virus-Checked: Checked by ClamAV on apache.org --1265908415-1107087233-1411525815=:61307 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi,=0A=0AI am worry about KMS and transparent encryption since there are qu= ite many bugs discovered after it got merged to branch-2. It gives us an i= mpression that the feature is not yet well tested. Indeed, transparent enc= ryption is a complicated feature which changes the core part of HDFS. It i= s not easy to get everything right.=0A=0A=0AFor HDFS-6584: Archival Storage= , it is a relatively simple and low risk feature. It introduces a new stor= age type ARCHIVE and the concept of block storage policy to HDFS. When a c= luster is configured with ARCHIVE storage, the blocks will be stored using = the appropriate storage types specified by storage policies assigned to the= files/directories. Cluster admin could disable the feature by simply not = configuring any storage type and not setting any storage policy as before. = As Suresh mentioned, HDFS-6584 is in the final stages to be merged to bra= nch-2.=0A=0ARegards,=0ATsz-Wo=0A=0A=0A=0AOn Wednesday, September 24, 2014 7= :00 AM, Suresh Srinivas wrote:=0A =0A=0A>=0A>=0A>I= actually would like to see both archival storage and single replica=0A>mem= ory writes to be in 2.6 release. Archival storage is in the final stages=0A= >of getting ready for branch-2 merge as Nicholas has already indicated on= =0A>the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of th= ese=0A>features are being in development for sometime.=0A>=0A>On Tue, Sep 2= 3, 2014 at 3:27 PM, Andrew Wang =0A>wrote:=0A>=0A= >> Hey Arun,=0A>>=0A>> Maybe we could do a quick run through of the Roadmap= wiki and add/retarget=0A>> things accordingly?=0A>>=0A>> I think the KMS a= nd transparent encryption are ready to go. We've got a=0A>> very few furthe= r bug fixes pending, but that's it.=0A>>=0A>> Two HDFS things that I think = probably won't make the end of the week are=0A>> archival storage (HDFS-658= 4) and single replica memory writes (HDFS-6581),=0A>> which I believe are u= nder the HSM banner. HDFS-6484 was just merged to=0A>> trunk and I think ne= eds a little more work before it goes into branch-2.=0A>> HDFS-6581 hasn't = even been merged to trunk yet, so seems a bit further off=0A>> yet.=0A>>=0A= >> Just my 2c as I did not work directly on these features. I just generall= y=0A>> shy away from shipping bits quite this fresh.=0A>>=0A>> Thanks,=0A>>= Andrew=0A>>=0A>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy wrote:=0A>>=0A>> > Looks like most of the content is in and hadoo= p-2.6 is shaping up nicely.=0A>> >=0A>> > I'll create branch-2.6 by end of = the week and we can go from there to=0A>> > stabilize it - hopefully in the= next few weeks.=0A>> >=0A>> > Thoughts?=0A>> >=0A>> > thanks,=0A>> > Arun= =0A>> >=0A>> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy =0A>> > wrote:=0A>> >=0A>> > > Folks,=0A>> > >=0A>> > > With hadoo= p-2.5 nearly done, it's time to start thinking ahead to=0A>> > > hadoop-2.6= .=0A>> > >=0A>> > > Currently, here is the Roadmap per the wiki:=0A>> > >= =0A>> > > =E2=80=A2 HADOOP=0A>> > > =E2=80=A2 Crede= ntial provider HADOOP-10607=0A>> > > =E2=80=A2 HDFS=0A>> > > = =E2=80=A2 Heterogeneous storage (Phase 2) - Support APIs for=0A>>= > using=0A>> > > storage tiers by the applications HDFS-5682=0A>> > > = =E2=80=A2 Memory as storage tier HDFS-5851=0A>> > > =E2= =80=A2 YARN=0A>> > > =E2=80=A2 Dynamic Resource Configurati= on YARN-291=0A>> > > =E2=80=A2 NodeManager Restart YARN-133= 6=0A>> > > =E2=80=A2 ResourceManager HA Phase 2 YARN-556=0A= >> > > =E2=80=A2 Support for admin-specified labels in YARN= YARN-796=0A>> > > =E2=80=A2 Support for automatic, shared = cache for YARN=0A>> > application=0A>> > > artifacts YARN-1492=0A>> > > = =E2=80=A2 Support NodeGroup layer topology on YARN YARN-18=0A>= > > > =E2=80=A2 Support for Docker containers in YARN YARN-= 1964=0A>> > > =E2=80=A2 YARN service registry YARN-913=0A>>= > >=0A>> > > My suspicion is, as is normal, some will make the cut and so= me won't.=0A>> > > Please do add/subtract from the list as appropriate. Ide= ally, it would=0A>> be=0A>> > > good to ship hadoop-2.6 in a 6-8 weeks (say= , October) to keep up a=0A>> > cadence.=0A>> > >=0A>> > > More importantly= , as we discussed previously, we'd like hadoop-2.6 to=0A>> be=0A>> > > the = *last* Apache Hadoop 2.x release which support JDK6. I'll start a=0A>> > > = discussion with other communities (HBase, Pig, Hive, Oozie etc.) and=0A>> s= ee=0A>> > > how they feel about this.=0A>> > >=0A>> > > thanks,=0A>> > > Ar= un=0A>> > >=0A>> > >=0A>> >=0A>> >=0A>> > --=0A>> >=0A>> > --=0A>> > Arun C= . Murthy=0A>> > Hortonworks Inc.=0A>> > http://hortonworks.com/=0A>> >=0A>>= > --=0A>> > CONFIDENTIALITY NOTICE=0A>> > NOTICE: This message is intended= for the use of the individual or entity=0A>> to=0A>> > which it is address= ed and may contain information that is confidential,=0A>> > privileged and = exempt from disclosure under applicable law. If the reader=0A>> > of this m= essage is not the intended recipient, you are hereby notified=0A>> that=0A>= > > any printing, copying, dissemination, distribution, disclosure or=0A>> = > forwarding of this communication is strictly prohibited. If you have=0A>>= > received this communication in error, please contact the sender=0A>> imm= ediately=0A>> > and delete it from your system. Thank You.=0A>> >=0A>>=0A>= =0A>=0A>=0A>-- =0A>http://hortonworks.com/download/=0A>=0A>-- =0A>CONFIDENT= IALITY NOTICE=0A>NOTICE: This message is intended for the use of the indivi= dual or entity to =0A>which it is addressed and may contain information tha= t is confidential, =0A>privileged and exempt from disclosure under applicab= le law. If the reader =0A>of this message is not the intended recipient, yo= u are hereby notified that =0A>any printing, copying, dissemination, distri= bution, disclosure or =0A>forwarding of this communication is strictly proh= ibited. If you have =0A>received this communication in error, please contac= t the sender immediately =0A>and delete it from your system. Thank You.=0A>= =0A> --1265908415-1107087233-1411525815=:61307--