hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: About 2.7.4 Release
Date Thu, 27 Jul 2017 21:48:03 GMT
Hey guys,

Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814

I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.

Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.

Thank you everybody for contributing.
--Konst

On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:

> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <shv.hadoop@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajisaka@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could  you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aajisaka@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> ________________________________
> From: Konstantin Shvachko <shv.hadoop@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Latest update on the links and filters. Here is the correct link for
> the
> filter:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> requestId=12340814
>
> Also updated: https://s.apache.org/Dzg4
>
> Had to do some Jira debugging. Sorry for confusion.
>
> Thanks,
> --Konstantin
>
> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Hey Akira,
>
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists
> only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jir
> a/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that
> filter.
> So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
>    https://issues.apache.org/jira/issues?filter=12340814
>   or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
> release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajisaka@apache.org>
> wrote:
>
> Hi Konstantin,
>
>
> Thank you for volunteering as release manager!
>
> Actually the original link works fine: https://s.apache.org/Dzg4
>
>
> I couldn't see the link. Maybe is it private filter?
>
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
> Hi Brahma Reddy Battula,
>
>
> Actually the original link works fine: https://s.apache.org/Dzg4
> Your link excludes closed and resolved issues, which needs
> backporting,
> and
> which we cannot reopen, as discussed in this thread earlier.
>
> Looked through the issues you proposed:
>
> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
> Seems like a new feature. It helps failover to standby node when
> primary
> is
> under heavy load, but it introduces new APIs, addresses, config
> parameters.
> And needs at least one follow up jira.
> Looks like a backward compatible change, though.
> Did you have a chance to run it in production?
>
> +1 on
> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>
> [HDFS-10987] Make Decommission less expensive when lot of ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-10987>
>
> issues.apache.org
> When user want to decommission a node which having 50M blocks ,it could
> hold the namesystem lock for long time.We've seen it is taking 36 sec.
> As
> we knew during this ...
>
>
>
> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>
>
> [HDFS-9902] Support different values of dfs.datanode.du ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-9902>
>
> issues.apache.org
> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
> The
> DISK size may be several ...
>
>
>
> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>
>
> Trash does not descent into child directories to check for ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-8312>
>
> issues.apache.org
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example: Run the following command to
> initialize directory ...
>
>
>
> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>
>
> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>
>
> https://issues.apache.org/jira/browse/HADOOP-14100>
>
> issues.apache.org
> Recently there was on vulnerability reported on jsch library. Its fixed
> in latest 0.1.54 version before CVE was made public.
> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>
>
>
>
> Added them to 2.7.4 release. You should see them via the above link
>
> now.
> Would be good if you could attach backport patches for some of them?
>
> Appreciate your help,
> --Konstantin
>
> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
> brahmareddy.battula@huawei.com> wrote:
>
>
> Looks following link is not correct..
>
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me
> know
> your thoughts on this.
>
> For Large Cluster:
> =============
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
> Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved"  per Storage Type
>
> For Security:
> =========
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
> descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
> jar
> to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
> mapreduce-dev@hadoop.apache.org
> ;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches
> which
> do not backport cleanly, to avoid having too many lying around.
> Otherwise
> we can directly attach to old ticket. If a clean backport does
> happen
> to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Great Zhe. Let's monitor the build.
>
>
> I marked all jiras I knew of for inclusion into 2.7.4 as I
> described
> before.
> Target Version/s: 2.7.4
> Label: release-blocker
>
> Here is the link to the list: https://s.apache.org/Dzg4 Please
> let
> me
> know if I missed anything.
> And feel free to pick up any. Most of backports are pretty
> straightforward, but not all.
>
> We can create tracking jiras for backporting if you need to run
> Jenkins on the patch (and since Allen does not allow reopening
> them).
> But I think the final patch should be attached to the original
> jira.
> Otherwise history will be hard to follow.
>
> Thanks,
> --Konstantin
>
> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zhz@apache.org> wrote:
>
> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>
> shv.hadoop@gmail.com>
>
>
> wrote:
>
>
> Hey guys,
>
>
> I and a few of my colleagues would like to help here and move
> 2.7.4
> release forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod
> hinted on managing the release. Vinod, if you still want the
> job /
> have bandwidth will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like
> to
>
> propose
>
>
>
> a
>
>
> few issues to be included in the release, see the list below.
>
> Those
> are mostly bug fixes and optimizations, which we already have in
> our
>
> internal
>
>
>
> branch and run in production. Plus one minor feature "node
>
>
> labeling", which we found very handy, when you have heterogeneous
>
> environments and mixed workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
> - set the target version to 2.7.4, and
> - add a new label "release-blocker"
> That way we will know issues targeted for the release without
> reopening them for backports.
>
> 4. I see quite a few people are interested in the release. With
> all
> the help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build
> for
>
> 2.7.4
>
>
>
> branch.
>
>
>
> Thanks,
> --Konstantin
>
> ==========  List of issue for 2.7.4  ===========
> ------ Backports
> HADOOP-12975 <https://issues.apache.org/jir
> a/browse/HADOOP-12975>.
> Add
>
> du
>
>
>
> jitters
>
>
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>
>
> batching
>
>
>
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
> NPE
>
>
> when applying AvailableSpaceBlockPlacementPolicy
>
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
> fsck
>
> removal
>
>
>
> of dot printing
>
>
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
> abort
> balancer if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
> skip
> small blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
> SLS
> simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
> SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
> SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
> SLS
>
> ----- Node labeling
> MAPREDUCE-6304
> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
> --
>
> Zhe Zhang
>
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message