hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Vavilapalli <vino...@hortonworks.com>
Subject Re: [VOTE] Release Apache Hadoop 2.6.1 RC1
Date Wed, 23 Sep 2015 18:52:21 GMT
Tx Eric, given the long cycle for 2.6.1, I’ll put MAPREDUCE-6334 into 2.6.2.

Thanks
+Vinod

On Sep 21, 2015, at 11:19 AM, Eric Payne <eric.payne1000@yahoo.com<mailto:eric.payne1000@yahoo.com>>
wrote:

Hi Vinod and everyone else who helped on this release!

Thank you very much for going through the work and effort to put this release together.

While doing my testing for Hadoop 2.6.1 RC1, I encountered the following issue:
[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput
shuffle handler - ASF JIRA<https://issues.apache.org/jira/browse/MAPREDUCE-6334>












[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException during ...<https://issues.apache.org/jira/browse/MAPREDUCE-6334>
We are seeing this happen when an NM's disk goes bad during the creation of map output(s)
the reducer's fetcher can read the shuffle header and reserve the memory


View on issues.apache.org<https://issues.apache.org/jira/browse/MAPREDUCE-6334>

Preview by Yahoo





This is actually something we ran across frequently when running 2.6. You may want to consider
pulling that in to 2.6.

Other than that, it looks fine. I did the following manual testing on a one-node cluster:
- successfully downloaded and compiled the code
- successfully ran streaming jobs
- successfully ran distributed shell jobs
-- as part of the distributed shell testing, I started the jobs with the '-keep_containers_across_application_attempts'
property set, killed the AM container, and verified that it would in fact keep the containers
running across AM restarts.
- sucessfully ran wordcount jobs.

Thank you,
-Eric Payne

________________________________
From: Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org>>
To: common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>; yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>;
hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>; mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>
Cc: vinodkv@apache.org<mailto:vinodkv@apache.org>; Sangjin Lee <sjlee0@gmail.com<mailto:sjlee0@gmail.com>>;
Akira AJISAKA <ajisakaa@oss.nttdata.co.jp<mailto:ajisakaa@oss.nttdata.co.jp>>
Sent: Wednesday, September 16, 2015 9:10 PM
Subject: [VOTE] Release Apache Hadoop 2.6.1 RC1

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
YARN-2890) that Jeff Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via repository.apache.org<http://repository.apache.org>
at
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
-  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
non-committed but desired patches. 2.6.1 is already big as is and is late
by any standard, we can definitely include them in the next release.
- The 2.6.1 wiki page [3] captures some (but not all) of the context of
the patches that we pushed in.
- Given the number of fixes pushed [4] in, we had to make a bunch of
changes to our original plan - we added a few improvements that helped us
backport patches easier (or in many cases made backports possible), and we
dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
HDFS-7611, HDFS-7843, HDFS-8850).
- I ran all the unit tests which (surprisingly?) passed. (Except for one,
which pointed out a missing fix HDFS-7552).

As discussed before [5]
- This release is the first point release after 2.6.0
- I’d like to use this as a starting release for 2.6.2 in a few weeks and
then follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread:
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets:
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
-Working-Notes
[4] List of 2.6.1 patches pushed:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release:
http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
- Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
2.6.1.
- The additional patches in RC1 that got into 2.6.1 all the way from 2.8
are NOT in 2.7.2 yet, this will be done as a followup.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message