Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9BADF7A88 for ; Thu, 25 Aug 2011 20:41:06 +0000 (UTC) Received: (qmail 77195 invoked by uid 500); 25 Aug 2011 20:41:06 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 77085 invoked by uid 500); 25 Aug 2011 20:41:05 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 77078 invoked by uid 99); 25 Aug 2011 20:41:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Aug 2011 20:41:04 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Aug 2011 20:41:01 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 7E6422388900 for ; Thu, 25 Aug 2011 20:40:41 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1161741 - in /hadoop/common/branches/branch-0.20-security-204: CHANGES.txt src/docs/releasenotes.html src/docs/relnotes.py Date: Thu, 25 Aug 2011 20:40:41 -0000 To: common-commits@hadoop.apache.org From: omalley@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110825204041.7E6422388900@eris.apache.org> Author: omalley Date: Thu Aug 25 20:40:40 2011 New Revision: 1161741 URL: http://svn.apache.org/viewvc?rev=1161741&view=rev Log: Update release notes for 0.20.204.0 Modified: hadoop/common/branches/branch-0.20-security-204/CHANGES.txt hadoop/common/branches/branch-0.20-security-204/src/docs/releasenotes.html hadoop/common/branches/branch-0.20-security-204/src/docs/relnotes.py Modified: hadoop/common/branches/branch-0.20-security-204/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-204/CHANGES.txt?rev=1161741&r1=1161740&r2=1161741&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20-security-204/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.20-security-204/CHANGES.txt Thu Aug 25 20:40:40 2011 @@ -1,6 +1,6 @@ Hadoop Change Log -Release 0.20.204.0 - 2011-8-19 +Release 0.20.204.0 - 2011-8-25 NEW FEATURES Modified: hadoop/common/branches/branch-0.20-security-204/src/docs/releasenotes.html URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-204/src/docs/releasenotes.html?rev=1161741&r1=1161740&r2=1161741&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20-security-204/src/docs/releasenotes.html (original) +++ hadoop/common/branches/branch-0.20-security-204/src/docs/releasenotes.html Thu Aug 25 20:40:40 2011 @@ -18,6 +18,21 @@

MAPREDUCE-2846. + Blocker bug reported by aw and fixed by owen.omalley (task, task-controller, tasktracker)
+ a small % of all tasks fail with DefaultTaskController
+
Fixed a race condition in writing the log index file that caused tasks to 'fail'.
MAPREDUCE-2804. + Blocker bug reported by aw and fixed by owen.omalley
+ "Creation of symlink to attempt log dir failed." message is not useful
+
Removed duplicate chmods of job log dir that were vulnerable to race conditions between tasks. Also improved the messages when the symlinks failed to be created.
MAPREDUCE-2651. + Major bug reported by bharathm and fixed by bharathm (task-controller)
+ Race condition in Linux Task Controller for job log directory creation
+
There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time.
MAPREDUCE-2621. Minor bug reported by sherri_chen and fixed by sherri_chen
TestCapacityScheduler fails with "Queue "q1" does not exist"
@@ -28,15 +43,20 @@ Add queue-level metrics 0.20-security branch

We would like to record and present the jobtracker metrics on a per-queue basis.
MAPREDUCE-2555. + Minor bug reported by tgraves and fixed by tgraves (tasktracker)
+ JvmInvalidate errors in the gridmix TT logs
+
Observing a lot of jvmValidate exceptions in TT logs for grid mix run

************************
2011-04-28 02:00:37,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 46121, call
statusUpdate(attempt_201104270735_5993_m_003305_0, org.apache.hadoop.mapred.MapTaskStatus@1840a9c,
org.apache.hadoop.mapred.JvmContext@1d4ab6b) from 127.0.0.1:50864: error: java.io.IOException: JvmValidate Failed.
Ignoring request from task: attempt_201104270735_5993_m_003305_0, with JvmId:
jvm_201104270735_5993_m_103399012gsbl20430: java.io.IOException: JvmValidate Failed. Ignoring request from task:
attempt_201104270735_5993_m_003305_0, with JvmId: jvm_201104270735_5993_m_103399012gsbl20430: --
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1386)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security. UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1384)

*********************
MAPREDUCE-2529. Major bug reported by tgraves and fixed by tgraves (tasktracker)
Recognize Jetty bug 1342 and handle it
-
We are seeing many instances of the Jetty-1342 (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to stop responding altogether, some fetches go through but a lot of them throw exceptions and eventually fail. The only way we have found to get the TT out of this state is to restart the TT. This jira is to catch this particular exception (or perhaps a configurable regex) and handle it in an automated way to either blacklist or shutdown the TT after seeing it a configurable number of them.

Added 2 new config parameters:

mapreduce.reduce.shuffle.catch.exception.stack.regex

mapreduce.reduce.shuffle.catch.exception.message.regex

MAPREDUCE-2524. Minor improvement reported by tgraves and fixed by tgraves (tasktracker)
Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
-
The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.

Added a new configuration option: mapreduce.reduce.shuffle.maxfetchfailures, and removed a no longer used option: mapred.reduce.copy.backoff.

MAPREDUCE-2514. Trivial bug reported by jeagles and fixed by jeagles (tasktracker)
@@ -56,7 +76,7 @@
MAPREDUCE-2479. Major improvement reported by revans2 and fixed by revans2 (tasktracker)
Backport MAPREDUCE-1568 to hadoop security branch
-

Added mapreduce.tasktracker.distributedcache.checkperiod to the task tracker that defined the period to wait while cleaning up the distributed cache. The default is 1 min.

MAPREDUCE-2456. Trivial improvement reported by naisbitt and fixed by naisbitt (jobtracker)
@@ -78,13 +98,23 @@ Fix FI build - broken after MR-2429

src/test/system/aop/org/apache/hadoop/mapred/TaskAspect.aj:72 [warning] advice defined in org.apache.hadoop.mapred.TaskAspect has not been applied [Xlint:adviceDidNotMatch]

After the fix in MR-2429, the call to ping in TaskAspect needs to be fixed.
MAPREDUCE-2429. + Major bug reported by acmurthy and fixed by sseth (tasktracker)
+ Check jvmid during task status report
+
Currently TT doens't check to ensure jvmid is relevant during communication with the Child via TaskUmbilicalProtocol.
MAPREDUCE-2418. + Minor bug reported by sseth and fixed by sseth
+ Errors not shown in the JobHistory servlet (specifically Counter Limit Exceeded)
+
Job error details are not displayed in the JobHistory servlet. e.g. Errors like 'Counter limit exceeded for a job'.
jobdetails.jsp has 'Failure Info', but this is missing in jobdetailshistory.jsp
MAPREDUCE-2415. - Major improvement reported by bharathm and fixed by bharathm (task-controller, tasktracker)
+ Major sub-task reported by bharathm and fixed by bharathm (task-controller, tasktracker)
Distribute TaskTracker userlogs onto multiple disks

Currently, userlogs directory in TaskTracker is placed under hadoop.log.dir like <hadoop.log.dir>/userlogs. I am proposing to spread these userlogs onto multiple configured mapred.local.dirs to strengthen TaskTracker reliability w.r.t disk failures.
MAPREDUCE-2413. - Major improvement reported by bharathm and fixed by ravidotg (task-controller, tasktracker)
+ Major sub-task reported by bharathm and fixed by ravidotg (task-controller, tasktracker)
TaskTracker should handle disk failures at both startup and runtime

At present, TaskTracker doesn't handle disk failures properly both at startup and runtime.

(1) Currently TaskTracker doesn't come up if any of the mapred-local-dirs is on a bad disk. TaskTracker should ignore that particular mapred-local-dir and start up and use only the remaining good mapred-local-dirs.
(2) If a disk goes bad while TaskTracker is running, currently TaskTracker doesn't do anything special. This results in either
(a) TaskTracker continues to "try to use that bad disk" and this results in lots of task failures and possibly job failures(because of multiple TTs having bad disks) and eventually these TTs getting graylisted for all jobs. And this needs manual restart of TT with modified configuration of mapred-local-dirs avoiding the bad disk. OR
(b) Health check script identifying the disk as bad and the TT gets blacklisted. And this also needs manual restart of TT with modified configuration of mapr ed-local-dirs avoiding the bad disk.

This JIRA is to make TaskTracker more fault-tolerant to disk failures solving (1) and (2). i.e. TT should start even if at least one of the mapred-local-dirs is on a good disk and TT should adjust its in-memory list of mapred-local-dirs and avoid using bad mapred-local-dirs.

Distributed Cache does not differentiate between file /archive for files with the same path

If a 'global' file is specified as a 'file' by one job - subsequent jobs cannot override this source file to be an 'archive' (until the TT cleans up it's cache or a TT restart).
The other way around as well -> 'archive' to 'file'

In case of an accidental submission using the wrong type - some of the tasks for the second job will end up seeing the source file as an archive, others as a file.

MAPREDUCE-2366. + Major bug reported by owen.omalley and fixed by dking (tasktracker)
+ TaskTracker can't retrieve stdout and stderr from web UI
+
Problem where the task browser UI can't retrieve the stdxxx printouts of streaming jobs that abend in the unix code, in the common case where the containing job doesn't reuse JVM's.
MAPREDUCE-2364. + Major bug reported by owen.omalley and fixed by devaraj (tasktracker)
+ Shouldn't hold lock on rjob while localizing resources.
+
There is a deadlock while localizing resources on the TaskTracker.
MAPREDUCE-2362. + Major bug reported by owen.omalley and fixed by roelofs (test)
+ Unit test failures: TestBadRecords and TestTaskTrackerMemoryManager
+
Fix unit-test failures: TestBadRecords (NPE due to rearranged MapTask code) and TestTaskTrackerMemoryManager (need hostname in output-string pattern).
MAPREDUCE-2360. + Major bug reported by owen.omalley and fixed by (client)
+ Pig fails when using non-default FileSystem
+
The job client strips the file system from the user's job jar, which causes breakage when it isn't the default file system.
MAPREDUCE-2359. + Major bug reported by owen.omalley and fixed by ramach
+ Distributed cache doesn't use non-default FileSystems correctly
+
We are passing fs.deafult.name as viewfs:/// in core site.xml on oozie server.
We have default name node in configuration also viewfs:///

We are using hdfs://path in our path for application.
Its giving following error:

IllegalArgumentException: Wrong FS:
hdfs://nn/user/strat_ci/oozie-oozi/0000002-110217014830452-oozie-oozi-W/hadoop1--map-reduce/map-reduce-launcher.jar,
expected: viewfs:/
MAPREDUCE-2358. + Major bug reported by owen.omalley and fixed by ramach
+ MapReduce assumes HDFS as the default filesystem
+
Mapred assumes hdfs as the default fs even when defined otherwise.
MAPREDUCE-2357. + Major bug reported by owen.omalley and fixed by vicaya (task)
+ When extending inputsplit (non-FileSplit), all exceptions are ignored
+
if you're using a custom RecordReader/InputFormat setup and using an
InputSplit that does NOT extend FileSplit, then any exceptions you throw in your RecordReader.nextKeyValue() function
are silently ignored.
MAPREDUCE-2356. + Major bug reported by owen.omalley and fixed by vicaya
+ A task succeeded even though there were errors on all attempts.
+
From Luke Lu:

Here is a summary of why the failed map task was considered "successful" (Thanks to Mahadev, Arun and Devaraj
for insightful discussions).

1. The map task was hanging BEFORE being initialized (probably in localization, but it doesn't matter in this case).
Its state is UNASSIGNED.

2. The jt decided to kill it due to timeout and scheduled a cleanup task on the same node.

3. The cleanup task has the same attempt id (by design.) but runs in a different JVM. Its initial state is
FAILED_UNCLEAN.

4. The JVM of the original attempt is getting killed, while proceeding to setupWorkDir and throwed an
IllegalStateException while FileSystem.getLocal, which causes taskFinal.taskCleanup being called in Child, and
triggered the NPE due to the task is not yet initialized (committer is null). Before the NPE, however it sent a
statusUpdate to TT, and in tip.reportProgress, changed the task state (currently FAILED_UNCLEAN) to UNASSIGNED.

5. The cleanup attempt succeeded and report done to TT. In tip.reportDone, the isCleanup() check returned false due to
the UNASSIGNED state and set the task state as SUCCEEDED.
MAPREDUCE-517. + Critical bug reported by acmurthy and fixed by acmurthy
+ The capacity-scheduler should assign multiple tasks per heartbeat
+
HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
MAPREDUCE-118. Blocker bug reported by amar_kamat and fixed by amareshwari (client)
Job.getJobID() will always return null
@@ -106,7 +181,7 @@
HDFS-2218. Blocker test reported by mattf and fixed by mattf (contrib/hdfsproxy, test)
Disable TestHdfsProxy.testHdfsProxyInterface in automated test suite for 0.20-security-204 release
-
To enable release of 0.20-security-204, despite the existence of unsolved bug HDFS-2217, remove this test case for 204. This is acceptable because HDFS-2217 is believed to be a bug in the test case and/or its interaction with the Hudson environment, not the HdfsProxy functionality.

To be fixed and restored for the next release.

Test case TestHdfsProxy.testHdfsProxyInterface has been temporarily disabled for this release, due to failure in the Hudson automated test environment.

HDFS-2057. Major bug reported by bharathm and fixed by bharathm (data-node)
@@ -128,11 +203,6 @@ TestHDFSServerPorts unit test failure - race condition in FSNamesystem.close() causes NullPointerException without serious consequence

In 20.204, TestHDFSServerPorts was observed to intermittently throw a NullPointerException. This only happens when FSNamesystem.close() is called, which means system termination for the Namenode, so this is not a serious bug for .204. TestHDFSServerPorts is more likely than normal execution to stimulate the race, because it runs two Namenodes in the same JVM, causing more interleaving and more potential to see a race condition.

The race is in FSNamesystem.close(), line 566, we have:
if (replthread != null) replthread.interrupt();
if (replmon != null) replmon = null;

Since the interrupted replthread is not waited on, there is a potential race condition with replmon being nulled before replthread is dead, but replthread references replmon in computeDatanodeWork() where the NullPointerException occurs.

The solution is either to wait on replthread or just don't null replmon. The latter is preferred, since none of th e sibling Namenode processing threads are waited on in close().

I'll attach a patch for .205.
HDFS-1836. - Major bug reported by hkdennis2k and fixed by bharathm (hdfs client)
- Thousand of CLOSE_WAIT socket
-
$ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT
4471

It is better if everything runs normal.
However, from time to time there are some "DataStreamer Exception: java.net.SocketTimeoutException" and "DFSClient.processDatanodeError(2507) | Error Recovery for" can be found from log file and the number of CLOSE_WAIT socket just keep increasing

The CLOSE_WAIT handles may remain for hours and days; then "Too many open file" some day.
HDFS-1822. Blocker bug reported by sureshms and fixed by sureshms (name-node)
Editlog opcodes overlap between 20 security and later releases
@@ -176,7 +246,7 @@
HDFS-1445. Major sub-task reported by mattf and fixed by mattf (data-node)
Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
-
It was a bit of a puzzle why we can do a full scan of a disk in about 30 seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes to do Upgrade replication via hardlinks. It turns out that the org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to Runtime.getRuntime().exec(), to utilize native filesystem hardlink capability. So it is forking a full-weight external process, and we call it on each individual file to be replicated.

As a simple check on the possible cost of this approach, I built a Perl test script (under Linux on a production-class datanode). Perl also uses a compiled and optimized p-code engine, and it has both native support for hardlinks and the ability to do "exec".
- A simple script to create 256,000 files in a directory tree organized like the Datanode, took 10 seconds to run.
- Replicating that directory tree using hardlinks, the same way as the Datanode, took 12 seconds using native hardlink support.
- The same replication using outcalls to exec, one per file, took 256 seconds!
- Batching the calls, and doing 'exec' once per directory instead of once per file, took 16 seconds.

Obviously, your mileage will vary based on the number of blocks per volume. A volume with less than about 4000 blocks will have only 65 directories. A volume with more than 4K and less than about 250K blocks will have 4200 directories (more or less). And there are two files per block (the data file and the .meta file). So the average number of files per directory may vary from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K files per volume, or an average of about 6:1. So this change may be expected to take it down from, say, 12 minutes per volume to 2.

Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Depends on prior integration with patch for HADOOP-7133.

HDFS-1377. Blocker bug reported by eli and fixed by eli (name-node)
@@ -256,8 +326,7 @@
HADOOP-6255. Major new feature reported by owen.omalley and fixed by eyang
Create an rpm integration project
-
We should be able to create RPMs for Hadoop releases.

Added RPM/DEB packages to build system.

Changes Since Hadoop 0.20.2

Modified: hadoop/common/branches/branch-0.20-security-204/src/docs/relnotes.py URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-204/src/docs/relnotes.py?rev=1161741&r1=1161740&r2=1161741&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20-security-204/src/docs/relnotes.py (original) +++ hadoop/common/branches/branch-0.20-security-204/src/docs/relnotes.py Thu Aug 25 20:40:40 2011 @@ -9,6 +9,7 @@ import csv import re +import subprocess import sys namePattern = re.compile(r' $[0-9]+$') @@ -32,7 +33,29 @@ def quoteHtmlChar(m): def quoteHtml(str): return re.sub(htmlSpecialPattern, quoteHtmlChar, str) -reader = csv.reader(sys.stdin, skipinitialspace=True) +def readReleaseNote(id, default): + cmd = ['jira.sh', '-s', 'https://issues.apache.org/jira', '-u', user, + '-p', password, '-a', 'getFieldValue', '--issue', id, '--field', + 'Release Note'] + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=sys.stderr) + lines = proc.stdout.readlines() + # throw away first line + if len(lines) < 2 or len(lines[1]) < 2: + return default + else: + return "\n".join(lines[1:])[1:-2] + +user = sys.argv[1] +password = sys.argv[2] +vers = sys.argv[3] + +cmd = ['jira.sh', '-s', 'https://issues.apache.org/jira', '-u', user, '-p', + password, '-a', 'getIssueList', '--search', + "project in (HADOOP,HDFS,MAPREDUCE) and fixVersion = '" + vers + + "' and resolution = Fixed"] +proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=sys.stderr) + +reader = csv.reader(proc.stdout, skipinitialspace=True) # throw away number of issues reader.next() @@ -52,6 +75,7 @@ components = columns.index('Components') print "

%s.\n' \ ' %s %s reported by %s and fixed by %s %s
\n' \ @@ -59,6 +83,6 @@ for row in reader: '
%s

\n"