Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 371FC200C02 for ; Fri, 20 Jan 2017 11:04:30 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 35A83160B55; Fri, 20 Jan 2017 10:04:30 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 07590160B48 for ; Fri, 20 Jan 2017 11:04:28 +0100 (CET) Received: (qmail 69981 invoked by uid 500); 20 Jan 2017 10:04:27 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 69940 invoked by uid 99); 20 Jan 2017 10:04:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jan 2017 10:04:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CDCFAC0221; Fri, 20 Jan 2017 10:04:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.218 X-Spam-Level: * X-Spam-Status: No, score=1.218 tagged_above=-999 required=6.31 tests=[FSL_HELO_BARE_IP_2=1.119, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id fbMhhzAD1b2m; Fri, 20 Jan 2017 10:04:24 +0000 (UTC) Received: from relayvx11b.securemail.intermedia.net (relayvx11b.securemail.intermedia.net [64.78.52.184]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DE4885F54F; Fri, 20 Jan 2017 10:04:23 +0000 (UTC) Received: from emg-ca-1-1.securemail.intermedia.net (localhost [127.0.0.1]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-1.localdomain (Postfix) with ESMTPS id C60A553EE2; Fri, 20 Jan 2017 02:04:06 -0800 (PST) Subject: Re: [Continued] [Release thread] 2.8.0 release activities MIME-Version: 1.0 x-echoworx-msg-id: d9c9d32a-5ad9-4f4f-bfa7-de500f57eddb x-echoworx-emg-received: Fri, 20 Jan 2017 02:04:06.712 -0800 x-echoworx-message-code-hashed: 4871fe1ceb8ef294343a6e874456d0fb899583684a1b8f204a86cd08106467f9 x-echoworx-action: delivered Received: from 10.254.155.14 ([10.254.155.14]) by emg-ca-1-1 (JAMES SMTP Server 2.3.2) with SMTP ID 882; Fri, 20 Jan 2017 02:04:06 -0800 (PST) Received: from MBX080-W4-CO-2.exch080.serverpod.net (unknown [10.224.117.102]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-1.localdomain (Postfix) with ESMTPS id 69FB153EE2; Fri, 20 Jan 2017 02:04:06 -0800 (PST) Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Fri, 20 Jan 2017 02:04:07 -0800 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1178.000; Fri, 20 Jan 2017 02:04:07 -0800 From: Junping Du To: "common-dev@hadoop.apache.org" , "hdfs-dev@hadoop.apache.org" , "mapreduce-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" Thread-Topic: [Continued] [Release thread] 2.8.0 release activities Thread-Index: AQHSSsVAd51TaAaIb02Die4ZuqO406DysmaA//+AUyqAAI6vAIAAk4lDgAle1QCAADP90oArTncQgBbxwD6AAc8tT4AAeCSw Date: Fri, 20 Jan 2017 10:04:06 +0000 Message-ID: <1484906644229.42595@hortonworks.com> References: <4B44247B-AFB2-4803-BDB0-251A8270F98C@apache.org> <2F9B8811-8240-454A-8599-64BE4B15CE2F@apache.org> <7CCEC640-5BCA-4ECC-976B-8630F4C0DD72@apache.org> <33841919-A63C-40BC-992D-9B98424BFBFC@hortonworks.com> <09457A47-B7F1-49C0-96CF-AA2AB07C6E2A@hortonworks.com> <5AD22D82-0F7C-4443-8FBB-B1295CB5707D@apache.org> <1480481443907.45579@hortonworks.com> <1480550805462.62858@hortonworks.com> <1480612149466.84413@hortonworks.com>,,<1481139119727.18762@hortonworks.com>,<1483523074647.66672@hortonworks.com>,<1484782008179.66066@hortonworks.com>,<1484880409796.52924@hortonworks.com> In-Reply-To: <1484880409796.52924@hortonworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [98.248.39.66] x-source-routing-agent: Processed Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable archived-at: Fri, 20 Jan 2017 10:04:30 -0000 Hi, I have successfully built the release bit on branch-2.8.0 by following:= https://wiki.apache.org/hadoop/HowToRelease step by step. However, when tr= y to close the "Staging Repositories" at Nexus page (https://repository.apa= che.org/#stagingRepositories), I found our repository - orgapachehadoop-105= 1 cannot be closed due to some signature validation failed and one Apache r= ule failed. I never met this problem before in previous releasing 2.6.3 and= 2.6.4. Is this related to our recent changes on release process/tools (doc= ker based)? Any ideas or thoughts on how to fix the problem? Thanks, Junping ________________________________________ From: Junping Du Sent: Thursday, January 19, 2017 6:46 PM To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev= @hadoop.apache.org; yarn-dev@hadoop.apache.org Cc: Varun Vasudev Subject: Re: [Continued] [Release thread] 2.8.0 release activities According to Varun's offline email, the security fixes has landed on branch= -2, 2.8 and 2.8.0 branch. I was kicking off a new RC build (RC1), and will publish it for vote soon. = In the mean time, please mark fix version as 2.8.1 for any new commits land= ed on branch-2.8, and don't commit anything to branch-2.8.0 at this moment.= Thanks! Cheers, Junping ________________________________________ From: Junping Du Sent: Wednesday, January 18, 2017 3:26 PM To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev= @hadoop.apache.org; yarn-dev@hadoop.apache.org Cc: Varun Vasudev Subject: Re: [Continued] [Release thread] 2.8.0 release activities Hi folks, In the passed one or two weeks, we found some new blockers get coming = on branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart w= ith work preserving enabled) and YARN-6072 (RM unable to start in secure mo= de). Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YA= RN-6068), and I was starting the RC build process since early this week. As= we have significant build tools/process change (docker based) since 2.8 co= mparing with 2.6/2.7, it takes me a while to get familiar with it and final= ly get a successful build on 2.8.0-RC0 last night. I already push RC0 tag into public which is prerequisite step before R= C voting. However, in the mean while, I was pinged by Varun Vasudev that t= here are a known vulnerability issues on container_executor get identified = and discussed in hadoop security email threads - looks like YARN-5704 fixed= part of it, but left part - the privilege escalation via /proc/self/enviro= n is not fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 althou= gh I haven't announced it public for vote yet. I will wait this issue get f= ixed to prepare a new release candidate. As RC0 tag cannot be reverted afte= r push into apache, our next release candidate will start from RC1. As I mentioned in early email, 2.8.0 is a very big release (2300+ commi= ts since 2.7.3) and I am glad that we are almost there. Thanks everyone for= being patient and contributing our release work. Please let me know if you= have more comments or suggestions. Thanks, Junping ________________________________________ From: Junping Du Sent: Wednesday, January 04, 2017 1:41 AM To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev= @hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: [Continued] [Release thread] 2.8.0 release activities Hi all Hadoopers, I just commit YARN-3866 which is the last blocker for branch-2.8, so s= ince tomorrow I will kick off process to prepare the first RC for our 2.8.0= release. This release will include at least 2374 commits that never landed= before in previous 2.x releases. Please check https://s.apache.org/RW5k fo= r details. There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but no= t marked with 2.8 (https://s.apache.org/RKti) that I need to double check a= ll are landed in branch-2.8 and fix the versions before kicking off the fir= st RC. Our progress is slightly behind my estimation weeks ago. However, cons= idering we just go through holidays and Jenkins cleanup issues linger on fo= r several projects (YARN, etc.), our achievement here is still great! Thank= s everyone for keep calm and carry on the help for the release. Kudos to Wa= ngda, Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on bloc= kers during this time. Also, special thanks to Vinod and Andrew for sharing= knowledge and practice (include scripts for auto version check) for releas= ing effort. Will try to prepare RC0 of 2.8 release for vote within this week. Stay= tuned! Thanks, Junping ________________________________________ From: Junping Du Sent: Wednesday, December 07, 2016 11:31 AM To: Akira Ajisaka; common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org= ; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: [Continued] [Release thread] 2.8.0 release activities Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for = several runs before last weekend, but it get failed for latest several runs= , probably affected by recently Jenkins down. However, from my recent manual kick off, it seems to be good again: https:/= /builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will keep= an eye for today's nightly run to see if it back to normal. Thanks, Junping ________________________________________ From: Akira Ajisaka Sent: Wednesday, December 07, 2016 12:12 AM To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev= @hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: [Continued] [Release thread] 2.8.0 release activities Thanks Junping and Andrew! HADOOP-2.8-JACC is not working well, so I manually kicked a job to compare 2.8 with 2.7.3. https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/art= ifact/target/compat-check/report.html Regards, Akira On 2016/12/02 2:08, Junping Du wrote: > Thanks Andrew! That's also a nice suggestion. I already create a similar = job: https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ fo= r 2.8 and kick off several runs manually. Will monitor incompatible status = from there. > > > Thanks, > > > Junping > > > ________________________________ > From: Andrew Wang > Sent: Wednesday, November 30, 2016 4:18 PM > To: Junping Du > Cc: Sangjin Lee; common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org= ; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod Vavila= palli; Jian He; Wangda Tan; sjlee@twitter.com > Subject: Re: [Continued] [Release thread] 2.8.0 release activities > > I recommend giving the JACC report another look. I set up a parameterized= jenkins job which you can trigger manually for branch-2.8: > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/ > > On Wed, Nov 30, 2016 at 4:06 PM, Junping Du > wrote: > Hi Sangjin and all, > > That sounds good. If anyone have priority fix to backport , please n= ominate it by following this email thread or ping me on JIRA directly. And = I agree that we should keep in mind that our bar for 2.8 release should be = high now as we want to move quickly on this release. Non-critical fixes can= wait for next one. > > Any other comments and suggestions? > > > Thanks, > > > Junping > > > ________________________________ > From: sjlee0@gmail.com > on behalf of Sangjin Lee > > Sent: Wednesday, November 30, 2016 3:24 PM > To: Junping Du > Cc: common-dev@hadoop.apache.org; hd= fs-dev@hadoop.apache.org; mapreduce-dev@= hadoop.apache.org; yarn-dev@hadoop.= apache.org; Vinod Vavilapalli; Jian He; = Wangda Tan; sjlee@twitter.com > Subject: Re: [Continued] [Release thread] 2.8.0 release activities > > Thanks for picking this up Junping! > > I heard on email threads and offline discussions that there may be intere= sts in porting some fixes from branch-2 to branch-2.8. I still feel that th= e bar should be pretty high to do that, but it might be good to have folks = suggest some critical fixes that need to be released sooner. Thoughts? > > On Tue, Nov 29, 2016 at 8:50 PM, Junping Du >= > wrote: > Hi folks, > Per asked by Vinod offline, I would like to continue the effort to p= ush 2.8 release out in next several weeks. The plan is as following: > - We only have 5 blockers and 3 critical issues (https://s.apache.or= g/6kwx) so far, and most of them are in good progress now. I put them all o= n apache ciwiki ( https://cwiki.apache.org/confluence/display/HADOOP/Hadoop= +2.8+Release) and will do daily monitor and update on these issues in next = couple of weeks. > - When all blocker/critical issues are gone (target EOD is Dec. 10),= I will push out all major/minor issues that is open. So if you are working= on some major issues and target for release in 2.8, please keep track of o= ur release status and your work progress. > - I will create RC immediately once our 2.8 target issues are clean = (especially for blocker and critical issues) for voting process. > > I hope we can make it within year 2016, best before Xmas holiday. An= d I need help from all of you. :) > > > Thanks, > > Junping > > ________________________________________ > From: Vinod Kumar Vavilapalli >> > Sent: Monday, July 25, 2016 2:57 PM > To: Jian He > Cc: mapreduce-dev@hadoop.apache.org>; yarn-dev@hadoop.apache.org>; hdfs-dev= @hadoop.apache.org>; common-dev@hadoop.apache.= org> > Subject: Re: [Release thread] 2.8.0 release activities > > Thanks for all that great help moving this release forward, Jian, Sangjin= , Wangda et al! Appreciate it. > > The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last= week. > > I only see 9 blocker / critical fixes pending for 2.8.0 - https://issues.= apache.org/jira/issues/?filter=3D12334985. Let=92s do this! > > Thanks > +Vinod > >> On May 11, 2016, at 6:04 PM, Jian He >> wrot= e: >> >> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs atten= tion for 2.8 >> >> MAPREDUCE-6288 >> YARN-1815 >> YARN-4685 >> YARN-4844 >> >> The rest are either improvements or long-standing issues and does not qu= alify release blocker, IMO. >> I think we=92ll try to get these 4 jiras in asap. The rest will be on be= st effort, resolve as much as possible and move them out if not resolved in= time. >> >> Jian >> >> On May 11, 2016, at 5:37 PM, Wangda Tan >>>> wrote: >> >> Sounds good to me :). >> >> Jian and I have looked at all existing 2.8.0 blockers and criticals toda= y. >> To me more than half of MR/YARN blockers/criticals of 2.8 should be move= d >> out. Left comments on these JIRAs asked original owners, plan to update >> target version of these JIRAs early next week. >> >> Will keep this thread updated. >> >> Thanks, >> Wangda >> >> >> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee >>>> wrote: >> >> How about this? I'll review the HADOOP/HDFS bugs in that list to come up >> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'l= l >> report the list here. Then folks can chime in if you agree >> >> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan? >> >> Thanks, >> Sangjin >> >> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan = >>>> wrote: >> >> +1, we should close such staled JIRAs to avoid doing unnecessary checks >> for >> every releases. >> >> I'm working on reviewing YARN/MR critical/blocker patches currently, it >> gonna very helpful if someone else can help with reviewing Common/HDFS >> JIRAs. >> >> Thanks, >> Wangda >> >> >> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee >>>> wrote: >> >> Where do we stand in terms of closing out blocker/critical issues for >> 2.8.0? I still see 50 open JIRAs in Vinod's list: >> https://issues.apache.org/jira/issues/?filter=3D12334985 >> >> But I see a lot of JIRAs with no patches or very stale patches. It >> would be >> a good exercise to come up with the list of JIRAs that we need to block >> 2.8.0 for and focus our attention on closing them out. Thoughts? >> >> Thanks, >> Sangjin >> >> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran >>> >> >> wrote: >> >> >> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli < >> vinodkv@apache.org>>>> >> wrote: >> >> We are not converging - there=92s still 58 more. I need help from the >> community in addressing / review 2.8.0 blockers. If folks can start >> with >> reviewing Patch available tickets, that=92ll be great. >> >> >> >> >> I'm still doing the s3a stuff, other people testing and reviewing this >> stuff welcome. >> >> in particular, I could do with others playing with this patch of mine, >> which adds counters and things into S3a, based on the azure >> instrumentation >> >> https://issues.apache.org/jira/browse/HADOOP-13028 >> >> >> >> >> >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org> > For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org> > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-dev-help@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-dev-help@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-dev-help@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-dev-help@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-dev-help@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org