Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CBDBE200C85 for ; Tue, 30 May 2017 09:08:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CA676160BC9; Tue, 30 May 2017 07:08:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C54A4160BC1 for ; Tue, 30 May 2017 09:08:09 +0200 (CEST) Received: (qmail 34191 invoked by uid 500); 30 May 2017 07:08:08 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 34149 invoked by uid 99); 30 May 2017 07:08:08 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 May 2017 07:08:08 +0000 Received: from mbpaa.local (KD111107190208.au-net.ne.jp [111.107.190.208]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id CBD9C1A031E; Tue, 30 May 2017 07:08:06 +0000 (UTC) Subject: Re: About 2.7.4 Release To: Rohith Sharma K S References: <8AD4EE147886274A8B495D6AF407DF698F25D8BE@blreml510-mbx.china.huawei.com> <978436dc-9a0c-1072-5092-bb62ec4a69f6@apache.org> Cc: Brahma Reddy Battula , Konstantin Shvachko , Hadoop Common , Hdfs-dev , "mapreduce-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" From: Akira Ajisaka Message-ID: Date: Tue, 30 May 2017 16:08:04 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit archived-at: Tue, 30 May 2017 07:08:11 -0000 Sure. If you want to edit the wiki, please tell me your ASF confluence account. -Akira On 2017/05/30 15:31, Rohith Sharma K S wrote: > Couple of more JIRAs need to be back ported for 2.7.4 release. These will > solve RM HA unstability issues. > https://issues.apache.org/jira/browse/YARN-5333 > https://issues.apache.org/jira/browse/YARN-5988 > https://issues.apache.org/jira/browse/YARN-6304 > > I will raise a JIRAs to back port it. > > @Akira , could you help to add these JIRAs into wiki? > > Thanks & Regards > Rohith Sharma K S > > On 29 May 2017 at 12:19, Akira Ajisaka wrote: > >> Created a page for 2.7.4 release. >> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4 >> >> If you want to edit this wiki, please ping me. >> >> Regards, >> Akira >> >> >> On 2017/05/23 4:42, Brahma Reddy Battula wrote: >> >>> Hi Konstantin Shvachko >>> >>> >>> how about creating a wiki page for 2.7.4 release status like 2.8 and >>> trunk in following link.?? >>> >>> >>> https://cwiki.apache.org/confluence/display/HADOOP >>> >>> >>> ________________________________ >>> From: Konstantin Shvachko >>> Sent: Saturday, May 13, 2017 3:58 AM >>> To: Akira Ajisaka >>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; >>> yarn-dev@hadoop.apache.org >>> Subject: Re: About 2.7.4 Release >>> >>> Latest update on the links and filters. Here is the correct link for the >>> filter: >>> https://issues.apache.org/jira/secure/IssueNavigator.jspa? >>> requestId=12340814 >>> >>> Also updated: https://s.apache.org/Dzg4 >>> >>> Had to do some Jira debugging. Sorry for confusion. >>> >>> Thanks, >>> --Konstantin >>> >>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko < >>> shv.hadoop@gmail.com> >>> wrote: >>> >>> Hey Akira, >>>> >>>> I didn't have private filters. Most probably Jira caches something. >>>> Your filter is in the right direction, but for some reason it lists only >>>> 22 issues, while mine has 29. >>>> It misses e.g. YARN-5543 >>> a/browse/YARN-5543> >>>> . >>>> >>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers", >>>> shared it with "everybody", and updated my link to point to that filter. >>>> So >>>> you can use any of the three methods below to get the correct list: >>>> 1. Go to https://s.apache.org/Dzg4 >>>> 2. Go to the filter via >>>> https://issues.apache.org/jira/issues?filter=12340814 >>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira >>>> 3. On Advanced issues search page paste this: >>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker >>>> AND "Target Version/s" = 2.7.4 >>>> >>>> Hope this solves the confusion for which issues are included. >>>> Please LMK if it doesn't, as it is important. >>>> >>>> Thanks, >>>> --Konstantin >>>> >>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka >>>> wrote: >>>> >>>> Hi Konstantin, >>>>> >>>>> Thank you for volunteering as release manager! >>>>> >>>>> Actually the original link works fine: https://s.apache.org/Dzg4 >>>>>> >>>>> I couldn't see the link. Maybe is it private filter? >>>>> >>>>> Here is a link I generated: https://s.apache.org/ehKy >>>>> This filter includes resolved issue and excludes fixversion == 2.7.4 >>>>> >>>>> Thanks and Regards, >>>>> Akira >>>>> >>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote: >>>>> >>>>> Hi Brahma Reddy Battula, >>>>>> >>>>>> Actually the original link works fine: https://s.apache.org/Dzg4 >>>>>> Your link excludes closed and resolved issues, which needs backporting, >>>>>> and >>>>>> which we cannot reopen, as discussed in this thread earlier. >>>>>> >>>>>> Looked through the issues you proposed: >>>>>> >>>>>> HDFS-9311 >>>>>> Seems like a new feature. It helps failover to standby node when >>>>>> primary >>>>>> is >>>>>> under heavy load, but it introduces new APIs, addresses, config >>>>>> parameters. >>>>>> And needs at least one follow up jira. >>>>>> Looks like a backward compatible change, though. >>>>>> Did you have a chance to run it in production? >>>>>> >>>>>> +1 on >>>>>> HDFS-10987 >>>>>> >>>>> [HDFS-10987] Make Decommission less expensive when lot of ...< >>> https://issues.apache.org/jira/browse/HDFS-10987> >>> issues.apache.org >>> When user want to decommission a node which having 50M blocks ,it could >>> hold the namesystem lock for long time.We've seen it is taking 36 sec. As >>> we knew during this ... >>> >>> >>> >>> HDFS-9902 >>>>>> >>>>> [HDFS-9902] Support different values of dfs.datanode.du ...< >>> https://issues.apache.org/jira/browse/HDFS-9902> >>> issues.apache.org >>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and >>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved. The >>> DISK size may be several ... >>> >>> >>> >>> HDFS-8312 >>>>>> >>>>> Trash does not descent into child directories to check for ...< >>> https://issues.apache.org/jira/browse/HDFS-8312> >>> issues.apache.org >>> HDFS trash does not descent into child directory to check if user has >>> permission to delete files. For example: Run the following command to >>> initialize directory ... >>> >>> >>> >>> HADOOP-14100 >>>>>> >>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...< >>> https://issues.apache.org/jira/browse/HADOOP-14100> >>> issues.apache.org >>> Recently there was on vulnerability reported on jsch library. Its fixed >>> in latest 0.1.54 version before CVE was made public. >>> https://cve.mitre.org/cgi-bin/cvename.cgi ... >>> >>> >>> >>> >>>>>> Added them to 2.7.4 release. You should see them via the above link >>>>>> now. >>>>>> Would be good if you could attach backport patches for some of them? >>>>>> >>>>>> Appreciate your help, >>>>>> --Konstantin >>>>>> >>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula < >>>>>> brahmareddy.battula@huawei.com> wrote: >>>>>> >>>>>> >>>>>> Looks following link is not correct.. >>>>>>> >>>>>>> https://s.apache.org/Dzg4 >>>>>>> >>>>>>> It should be like following..? >>>>>>> >>>>>>> https://s.apache.org/wi3U >>>>>>> >>>>>>> >>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me >>>>>>> know >>>>>>> your thoughts on this. >>>>>>> >>>>>>> For Large Cluster: >>>>>>> ============= >>>>>>> >>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol >>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission >>>>>>> Expensive when lot's of blocks are present >>>>>>> >>>>>>> https://issues.apache.org/jira/browse/HDFS-9902=== >>>>>>> "dfs.datanode.du.reserved" per Storage Type >>>>>>> >>>>>>> For Security: >>>>>>> ========= >>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not >>>>>>> descent >>>>>>> into child directories to check for permission >>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar >>>>>>> to >>>>>>> latest version to fix vulnerability in old versions >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Brahma Reddy Battula >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID] >>>>>>> Sent: 06 May 2017 02:40 >>>>>>> To: Konstantin Shvachko >>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; >>>>>>> mapreduce-dev@hadoop.apache.org >>>>>>> ; >>>>>>> yarn-dev@hadoop.apache.org >>>>>>> Subject: Re: About 2.7.4 Release >>>>>>> >>>>>>> List LGTM Konstantin! >>>>>>> >>>>>>> Let's say that we will only create a new tracking JIRA for patches >>>>>>> which >>>>>>> do not backport cleanly, to avoid having too many lying around. >>>>>>> Otherwise >>>>>>> we can directly attach to old ticket. If a clean backport does happen >>>>>>> to >>>>>>> break a test the nightly build will help us catch it. >>>>>>> >>>>>>> Erik >>>>>>> >>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko < >>>>>>> shv.hadoop@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Great Zhe. Let's monitor the build. >>>>>>> >>>>>>>> >>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I described >>>>>>>> before. >>>>>>>> Target Version/s: 2.7.4 >>>>>>>> Label: release-blocker >>>>>>>> >>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please let >>>>>>>> me >>>>>>>> know if I missed anything. >>>>>>>> And feel free to pick up any. Most of backports are pretty >>>>>>>> straightforward, but not all. >>>>>>>> >>>>>>>> We can create tracking jiras for backporting if you need to run >>>>>>>> Jenkins on the patch (and since Allen does not allow reopening them). >>>>>>>> But I think the final patch should be attached to the original jira. >>>>>>>> Otherwise history will be hard to follow. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> --Konstantin >>>>>>>> >>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang wrote: >>>>>>>> >>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM. >>>>>>>> >>>>>>>>> >>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests): >>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/ >>>>>>>>> >>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko < >>>>>>>>> >>>>>>>>> shv.hadoop@gmail.com> >>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hey guys, >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I and a few of my colleagues would like to help here and move 2.7.4 >>>>>>>>>> release forward. A few points in this regard. >>>>>>>>>> >>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod >>>>>>>>>> hinted on managing the release. Vinod, if you still want the job / >>>>>>>>>> have bandwidth will be happy to work with you. >>>>>>>>>> Otherwise I am glad to volunteer as the release manager. >>>>>>>>>> >>>>>>>>>> 2. In addition to current blockers and criticals, I would like to >>>>>>>>>> >>>>>>>>>> propose >>>>>>>>> >>>>>>>> >>>>>>>> a >>>>>>>>> >>>>>>>>>> few issues to be included in the release, see the list below. Those >>>>>>>>>> are mostly bug fixes and optimizations, which we already have in >>>>>>>>>> our >>>>>>>>>> >>>>>>>>>> internal >>>>>>>>> >>>>>>>> >>>>>>>> branch and run in production. Plus one minor feature "node >>>>>>>>> >>>>>>>>>> labeling", which we found very handy, when you have heterogeneous >>>>>>>>>> environments and mixed workloads, like MR and Spark. >>>>>>>>>> >>>>>>>>>> 3. For marking issues for the release I propose to >>>>>>>>>> - set the target version to 2.7.4, and >>>>>>>>>> - add a new label "release-blocker" >>>>>>>>>> That way we will know issues targeted for the release without >>>>>>>>>> reopening them for backports. >>>>>>>>>> >>>>>>>>>> 4. I see quite a few people are interested in the release. With all >>>>>>>>>> the help I think we can target to release by the end of May. >>>>>>>>>> >>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build >>>>>>>>>> for >>>>>>>>>> >>>>>>>>>> 2.7.4 >>>>>>>>> >>>>>>>> >>>>>>>> branch. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> --Konstantin >>>>>>>>>> >>>>>>>>>> ========== List of issue for 2.7.4 =========== >>>>>>>>>> ------ Backports >>>>>>>>>> HADOOP-12975 . >>>>>>>>>> Add >>>>>>>>>> >>>>>>>>>> du >>>>>>>>> >>>>>>>> >>>>>>>> jitters >>>>>>>>> >>>>>>>>>> HDFS-9710 . IBR >>>>>>>>>> >>>>>>>>>> batching >>>>>>>>> >>>>>>>> >>>>>>>> HDFS-10715 . NPE >>>>>>>>> >>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy >>>>>>>>>> HDFS-2538 . fsck >>>>>>>>>> >>>>>>>>>> removal >>>>>>>>> >>>>>>>> >>>>>>>> of dot printing >>>>>>>>> >>>>>>>>>> HDFS-8131 . >>>>>>>>>> space-balanced >>>>>>>>>> policy for balancer >>>>>>>>>> HDFS-8549 . abort >>>>>>>>>> balancer if upgrade in progress >>>>>>>>>> HDFS-9412 . skip >>>>>>>>>> small blocks in getBlocks >>>>>>>>>> >>>>>>>>>> YARN-1471 . SLS >>>>>>>>>> simulator >>>>>>>>>> YARN-4302 . SLS >>>>>>>>>> YARN-4367 . SLS >>>>>>>>>> YARN-4612 . SLS >>>>>>>>>> >>>>>>>>>> ----- Node labeling >>>>>>>>>> MAPREDUCE-6304 >>>>>>>>>> >>>>>>>>>> YARN-2943 >>>>>>>>>> YARN-4109 >>>>>>>>>> YARN-4140 >>>>>>>>>> YARN-4250 >>>>>>>>>> YARN-4925 >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>> Zhe Zhang >>>>>>>>> Apache Hadoop Committer >>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org >> For additional commands, e-mail: common-dev-help@hadoop.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org