Return-Path: X-Original-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 67AB3472E for ; Tue, 21 Jun 2011 20:27:01 +0000 (UTC) Received: (qmail 1374 invoked by uid 500); 21 Jun 2011 20:27:01 -0000 Delivered-To: apmail-hadoop-mapreduce-commits-archive@hadoop.apache.org Received: (qmail 1330 invoked by uid 500); 21 Jun 2011 20:27:00 -0000 Mailing-List: contact mapreduce-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-commits@hadoop.apache.org Received: (qmail 1322 invoked by uid 99); 21 Jun 2011 20:27:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:27:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:26:53 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 767D423888FD; Tue, 21 Jun 2011 20:26:31 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1138166 - in /hadoop/common/branches/MR-279/mapreduce: ./ mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/ mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/ mr-client/hadoo... Date: Tue, 21 Jun 2011 20:26:31 -0000 To: mapreduce-commits@hadoop.apache.org From: llu@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110621202631.767D423888FD@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: llu Date: Tue Jun 21 20:26:30 2011 New Revision: 1138166 URL: http://svn.apache.org/viewvc?rev=1138166&view=rev Log: MAPREDUCE-2611. Fix counters, finish times etc in job history. (Siddharth Seth via llu) Added: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedPeriodicStatsAccumulator.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java Modified: hadoop/common/branches/MR-279/mapreduce/CHANGES.txt hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java hadoop/common/branches/MR-279/mapreduce/yarn/yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java hadoop/common/branches/MR-279/mapreduce/yarn/yarn-server/yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java Modified: hadoop/common/branches/MR-279/mapreduce/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/CHANGES.txt?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/CHANGES.txt (original) +++ hadoop/common/branches/MR-279/mapreduce/CHANGES.txt Tue Jun 21 20:26:30 2011 @@ -5,6 +5,9 @@ Trunk (unreleased changes) MAPREDUCE-279 + MAPREDUCE-2611. Fix counters, finish times etc. in job history. + (Siddharth Seth via llu) + Fixing scheduling deadlock in AM because of incorrect headRoom values from RM. The bug happens when AM releases containers and RM decrements current memory usage twice for all those containers. (vinodkv) Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java Tue Jun 21 20:26:30 2011 @@ -126,9 +126,8 @@ public class MapReduceChildJVM { } } - // TODO: Put a random pid in env for now. - // Long term we will need to get it from the Child - env.put("JVM_PID", "12344"); + //This should not be set here (If an OS check is requied. moved to ContainerLuanch) + // env.put("JVM_PID", "`echo $$`"); env.put(Constants.STDOUT_LOGFILE_ENV, getTaskLogFile(containerLogDir, TaskLog.LogName.STDOUT).toString()); Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java Tue Jun 21 20:26:30 2011 @@ -181,7 +181,7 @@ public class TaskAttemptListenerImpl ext TypeConverter.toYarn(taskAttemptID); taskHeartbeatHandler.receivedPing(attemptID); - + //Ignorable TaskStatus? - since a task will send a LastStatusUpdate context.getEventHandler().handle( new TaskAttemptEvent(attemptID, TaskAttemptEventType.TA_COMMIT_PENDING)); @@ -314,6 +314,24 @@ public class TaskAttemptListenerImpl ext taskAttemptStatus.counters = TypeConverter.toYarn(taskStatus.getCounters()); + // Map Finish time set by the task (map only) + if (taskStatus.getIsMap() && taskStatus.getMapFinishTime() != 0) { + taskAttemptStatus.mapFinishTime = taskStatus.getMapFinishTime(); + } + + // Shuffle Finish time set by the task (reduce only). + if (!taskStatus.getIsMap() && taskStatus.getShuffleFinishTime() != 0) { + taskAttemptStatus.shuffleFinishTime = taskStatus.getShuffleFinishTime(); + } + + // Sort finish time set by the task (reduce only). + if (!taskStatus.getIsMap() && taskStatus.getSortFinishTime() != 0) { + taskAttemptStatus.sortFinishTime = taskStatus.getSortFinishTime(); + } + + // Not Setting the task state. Used by speculation - will be set in TaskAttemptImpl + //taskAttemptStatus.taskState = TypeConverter.toYarn(taskStatus.getRunState()); + //set the fetch failures if (taskStatus.getFetchFailedMaps() != null && taskStatus.getFetchFailedMaps().size() > 0) { @@ -325,8 +343,8 @@ public class TaskAttemptListenerImpl ext } } - // Task sends the information about the nextRecordRange to the TT - + // Task sends the information about the nextRecordRange to the TT + // TODO: The following are not needed here, but needed to be set somewhere inside AppMaster. // taskStatus.getRunState(); // Set by the TT/JT. Transform into a state TODO // taskStatus.getStartTime(); // Used to be set by the TaskTracker. This should be set by getTask(). Added: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedPeriodicStatsAccumulator.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedPeriodicStatsAccumulator.java?rev=1138166&view=auto ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedPeriodicStatsAccumulator.java (added) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedPeriodicStatsAccumulator.java Tue Jun 21 20:26:30 2011 @@ -0,0 +1,15 @@ +package org.apache.hadoop.mapred; + +//Workaround for PeriodicStateAccumulator being package access +public class WrappedPeriodicStatsAccumulator { + + private PeriodicStatsAccumulator real; + + public WrappedPeriodicStatsAccumulator(PeriodicStatsAccumulator real) { + this.real = real; + } + + public void extend(double newProgress, int newValue) { + real.extend(newProgress, newValue); + } +} Added: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java?rev=1138166&view=auto ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java (added) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java Tue Jun 21 20:26:30 2011 @@ -0,0 +1,52 @@ +package org.apache.hadoop.mapred; + +// Workaround for ProgressSplitBlock being package access +public class WrappedProgressSplitsBlock extends ProgressSplitsBlock { + + public static final int DEFAULT_NUMBER_PROGRESS_SPLITS = 12; + + private WrappedPeriodicStatsAccumulator wrappedProgressWallclockTime; + private WrappedPeriodicStatsAccumulator wrappedProgressCPUTime; + private WrappedPeriodicStatsAccumulator wrappedProgressVirtualMemoryKbytes; + private WrappedPeriodicStatsAccumulator wrappedProgressPhysicalMemoryKbytes; + + public WrappedProgressSplitsBlock(int numberSplits) { + super(numberSplits); + } + + public int[][] burst() { + return super.burst(); + } + + public WrappedPeriodicStatsAccumulator getProgressWallclockTime() { + if (wrappedProgressWallclockTime == null) { + wrappedProgressWallclockTime = new WrappedPeriodicStatsAccumulator( + progressWallclockTime); + } + return wrappedProgressWallclockTime; + } + + public WrappedPeriodicStatsAccumulator getProgressCPUTime() { + if (wrappedProgressCPUTime == null) { + wrappedProgressCPUTime = new WrappedPeriodicStatsAccumulator( + progressCPUTime); + } + return wrappedProgressCPUTime; + } + + public WrappedPeriodicStatsAccumulator getProgressVirtualMemoryKbytes() { + if (wrappedProgressVirtualMemoryKbytes == null) { + wrappedProgressVirtualMemoryKbytes = new WrappedPeriodicStatsAccumulator( + progressVirtualMemoryKbytes); + } + return wrappedProgressVirtualMemoryKbytes; + } + + public WrappedPeriodicStatsAccumulator getProgressPhysicalMemoryKbytes() { + if (wrappedProgressPhysicalMemoryKbytes == null) { + wrappedProgressPhysicalMemoryKbytes = new WrappedPeriodicStatsAccumulator( + progressPhysicalMemoryKbytes); + } + return wrappedProgressPhysicalMemoryKbytes; + } +} \ No newline at end of file Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java Tue Jun 21 20:26:30 2011 @@ -103,6 +103,7 @@ class YarnChild { // report non-pid to application master JvmContext context = new JvmContext(jvmId, "-1000"); + LOG.debug("PID: " + System.getenv().get("JVM_PID")); Task task = null; UserGroupInformation childUGI = null; Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java Tue Jun 21 20:26:30 2011 @@ -70,4 +70,5 @@ public interface AMConstants { public static final String NODE_BLACKLISTING_ENABLE = MRConstants.YARN_MR_PREFIX + "node.blacklisting.enable"; + } Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java Tue Jun 21 20:26:30 2011 @@ -23,6 +23,7 @@ import java.util.List; import org.apache.hadoop.mapreduce.v2.api.records.Counters; import org.apache.hadoop.mapreduce.v2.api.records.Phase; import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId; +import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptState; public class TaskAttemptStatusUpdateEvent extends TaskAttemptEvent { @@ -52,5 +53,9 @@ public class TaskAttemptStatusUpdateEven public Phase phase; public long outputSize; public List fetchFailedMaps; + public long mapFinishTime; + public long shuffleFinishTime; + public long sortFinishTime; + public TaskAttemptState taskState; } } Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java Tue Jun 21 20:26:30 2011 @@ -452,6 +452,33 @@ public class JobImpl implements org.apac } } + private Counters getTypeCounters(Set taskIds) { + Counters counters = newCounters(); + for (TaskId taskId : taskIds) { + Task task = tasks.get(taskId); + incrAllCounters(counters, task.getCounters()); + } + return counters; + } + + private Counters getMapCounters() { + readLock.lock(); + try { + return getTypeCounters(mapTasks); + } finally { + readLock.unlock(); + } + } + + private Counters getReduceCounters() { + readLock.lock(); + try { + return getTypeCounters(reduceTasks); + } finally { + readLock.unlock(); + } + } + public static Counters newCounters() { Counters counters = RecordFactoryProvider.getRecordFactory(null) .newRecordInstance(Counters.class); @@ -1066,8 +1093,8 @@ public class JobImpl implements org.apac job.oldJobId, job.finishTime, job.succeededMapTaskCount, job.succeededReduceTaskCount, job.failedMapTaskCount, job.failedReduceTaskCount, - TypeConverter.fromYarn(job.getCounters()), //TODO replace with MapCounters - TypeConverter.fromYarn(job.getCounters()), //TODO replace with ReduceCoutners + TypeConverter.fromYarn(job.getMapCounters()), + TypeConverter.fromYarn(job.getReduceCounters()), TypeConverter.fromYarn(job.getCounters())); return jfe; } Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java Tue Jun 21 20:26:30 2011 @@ -43,13 +43,16 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.DataOutputBuffer; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.MapReduceChildJVM; +import org.apache.hadoop.mapred.ProgressSplitsBlock; import org.apache.hadoop.mapred.ShuffleHandler; import org.apache.hadoop.mapred.Task; import org.apache.hadoop.mapred.TaskAttemptContextImpl; import org.apache.hadoop.mapred.WrappedJvmID; +import org.apache.hadoop.mapred.WrappedProgressSplitsBlock; import org.apache.hadoop.mapreduce.MRJobConfig; import org.apache.hadoop.mapreduce.OutputCommitter; import org.apache.hadoop.mapreduce.TaskAttemptContext; +import org.apache.hadoop.mapreduce.TaskCounter; import org.apache.hadoop.mapreduce.TypeConverter; import org.apache.hadoop.mapreduce.filecache.DistributedCache; import org.apache.hadoop.mapreduce.jobhistory.JobHistoryEvent; @@ -60,6 +63,8 @@ import org.apache.hadoop.mapreduce.jobhi import org.apache.hadoop.mapreduce.security.TokenCache; import org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier; import org.apache.hadoop.mapreduce.v2.MRConstants; +import org.apache.hadoop.mapreduce.v2.api.records.Counter; +import org.apache.hadoop.mapreduce.v2.api.records.CounterGroup; import org.apache.hadoop.mapreduce.v2.api.records.Counters; import org.apache.hadoop.mapreduce.v2.api.records.Phase; import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId; @@ -88,6 +93,7 @@ import org.apache.hadoop.mapreduce.v2.ap import org.apache.hadoop.mapreduce.v2.app.rm.ContainerRequestEvent; import org.apache.hadoop.mapreduce.v2.app.speculate.SpeculatorEvent; import org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanupEvent; +import org.apache.hadoop.mapreduce.v2.jobhistory.JHConfig; import org.apache.hadoop.mapreduce.v2.util.MRApps; import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.Credentials; @@ -124,6 +130,7 @@ public abstract class TaskAttemptImpl im EventHandler { private static final Log LOG = LogFactory.getLog(TaskAttemptImpl.class); + private static final long MEMORY_SPLITS_RESOLUTION = 1024; //TODO Make configurable? private final static RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); protected final Configuration conf; @@ -147,6 +154,7 @@ public abstract class TaskAttemptImpl im private long launchTime; private long finishTime; + private WrappedProgressSplitsBlock progressSplitBlock; private static final CleanupContainerTransition CLEANUP_CONTAINER_TRANSITION = new CleanupContainerTransition(); @@ -900,11 +908,82 @@ public abstract class TaskAttemptImpl im TypeConverter.fromYarn(taskAttempt.attemptId.getTaskId().getTaskType()), attemptState.toString(), taskAttempt.finishTime, taskAttempt.containerMgrAddress == null ? "UNKNOWN" : taskAttempt.containerMgrAddress, - taskAttempt.reportedStatus.diagnosticInfo.toString()); - //TODO Different constructor - allSplits + taskAttempt.reportedStatus.diagnosticInfo.toString(), + taskAttempt.getProgressSplitBlock().burst()); return tauce; } + private WrappedProgressSplitsBlock getProgressSplitBlock() { + readLock.lock(); + try { + if (progressSplitBlock == null) { + progressSplitBlock = new WrappedProgressSplitsBlock(conf.getInt( + JHConfig.JOBHISTORY_TASKPROGRESS_NUMBER_SPLITS_KEY, + WrappedProgressSplitsBlock.DEFAULT_NUMBER_PROGRESS_SPLITS)); + } + return progressSplitBlock; + } finally { + readLock.unlock(); + } + } + + private void updateProgressSplits() { + double newProgress = reportedStatus.progress; + Counters counters = reportedStatus.counters; + if (counters == null) + return; + + WrappedProgressSplitsBlock splitsBlock = getProgressSplitBlock(); + if (splitsBlock != null) { + long now = clock.getTime(); + long start = getLaunchTime(); // TODO Ensure not 0 + + if (start != 0 && now - start <= Integer.MAX_VALUE) { + splitsBlock.getProgressWallclockTime().extend(newProgress, + (int) (now - start)); + } + + // TODO Fix the Counter API + CounterGroup cpuCounterGroup = counters + .getCounterGroup(TaskCounter.CPU_MILLISECONDS.getDeclaringClass() + .getName()); + if (cpuCounterGroup != null) { + Counter cpuCounter = cpuCounterGroup + .getCounter(TaskCounter.CPU_MILLISECONDS.name()); + if (cpuCounter != null && cpuCounter.getValue() <= Integer.MAX_VALUE) { + splitsBlock.getProgressCPUTime().extend(newProgress, + (int) cpuCounter.getValue()); + } + } + + // TODO Fix the Counter API + CounterGroup vbCounterGroup = counters + .getCounterGroup(TaskCounter.VIRTUAL_MEMORY_BYTES.getDeclaringClass() + .getName()); + if (vbCounterGroup != null) { + Counter virtualBytes = vbCounterGroup + .getCounter(TaskCounter.VIRTUAL_MEMORY_BYTES.name()); + if (virtualBytes != null) { + splitsBlock.getProgressVirtualMemoryKbytes().extend(newProgress, + (int) (virtualBytes.getValue() / (MEMORY_SPLITS_RESOLUTION))); + } + } + + // TODO Fix the Counter API + CounterGroup pbCounterGroup = counters + .getCounterGroup(TaskCounter.PHYSICAL_MEMORY_BYTES + .getDeclaringClass().getName()); + if (pbCounterGroup != null) { + Counter physicalBytes = pbCounterGroup + .getCounter(TaskCounter.PHYSICAL_MEMORY_BYTES.name()); + if (physicalBytes != null) { + splitsBlock.getProgressPhysicalMemoryKbytes().extend(newProgress, + (int) (physicalBytes.getValue() / (MEMORY_SPLITS_RESOLUTION))); + } + } + } + } + private static class RequestContainerTransition implements SingleArcTransition { private final boolean rescheduled; @@ -1150,10 +1229,11 @@ public abstract class TaskAttemptImpl im new MapAttemptFinishedEvent(TypeConverter.fromYarn(attemptId), TypeConverter.fromYarn(attemptId.getTaskId().getTaskType()), state.toString(), - finishTime, //TODO TaskAttemptStatus changes. MapFinishTime + this.reportedStatus.mapFinishTime, finishTime, this.containerMgrAddress == null ? "UNKNOWN" : this.containerMgrAddress, - state.toString(), //TODO state is a progress string. - TypeConverter.fromYarn(getCounters()),null); + this.reportedStatus.stateString, + TypeConverter.fromYarn(getCounters()), + getProgressSplitBlock().burst()); eventHandler.handle( new JobHistoryEvent(attemptId.getTaskId().getJobId(), mfe)); } else { @@ -1161,11 +1241,12 @@ public abstract class TaskAttemptImpl im new ReduceAttemptFinishedEvent(TypeConverter.fromYarn(attemptId), TypeConverter.fromYarn(attemptId.getTaskId().getTaskType()), state.toString(), - finishTime, //TODO TaskAttemptStatus changes. ShuffleFinishTime - finishTime, //TODO TaskAttemptStatus changes. SortFinishTime + this.reportedStatus.shuffleFinishTime, + this.reportedStatus.sortFinishTime, finishTime, this.containerMgrAddress == null ? "UNKNOWN" : this.containerMgrAddress, - state.toString(), - TypeConverter.fromYarn(getCounters()),null); + this.reportedStatus.stateString, + TypeConverter.fromYarn(getCounters()), + getProgressSplitBlock().burst()); eventHandler.handle( new JobHistoryEvent(attemptId.getTaskId().getJobId(), rfe)); } @@ -1252,6 +1333,7 @@ public abstract class TaskAttemptImpl im .getReportedTaskAttemptStatus(); // Now switch the information in the reportedStatus taskAttempt.reportedStatus = newReportedStatus; + taskAttempt.reportedStatus.taskState = taskAttempt.getState(); // send event to speculator about the reported status taskAttempt.eventHandler.handle @@ -1260,6 +1342,7 @@ public abstract class TaskAttemptImpl im //add to diagnostic taskAttempt.addDiagnosticInfo(newReportedStatus.diagnosticInfo); + taskAttempt.updateProgressSplits(); //if fetch failures are present, send the fetch failure event to job //this only will happen in reduce attempt type @@ -1289,6 +1372,7 @@ public abstract class TaskAttemptImpl im result.diagnosticInfo = new String(""); result.phase = Phase.STARTING; result.stateString = new String("NEW"); + result.taskState = TaskAttemptState.NEW; Counters counters = recordFactory.newRecordInstance(Counters.class); // counters.groups = new HashMap(); result.counters = counters; Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java Tue Jun 21 20:26:30 2011 @@ -331,7 +331,7 @@ public class RecoveryService extends Com taskAttemptStatus.id = yarnAttemptID; taskAttemptStatus.progress = 1.0f; taskAttemptStatus.diagnosticInfo = ""; - taskAttemptStatus.stateString = attemptInfo.getState(); + taskAttemptStatus.stateString = attemptInfo.getTaskStatus(); // taskAttemptStatus.outputSize = attemptInfo.getOutputSize(); taskAttemptStatus.phase = Phase.CLEANUP; org.apache.hadoop.mapreduce.Counters cntrs = attemptInfo.getCounters(); Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java Tue Jun 21 20:26:30 2011 @@ -298,7 +298,7 @@ public class DefaultSpeculator extends A */ protected void statusUpdate(TaskAttemptStatus reportedStatus, long timestamp) { - String stateString = reportedStatus.stateString.toString(); + String stateString = reportedStatus.taskState.toString(); TaskAttemptId attemptID = reportedStatus.id; TaskId taskID = attemptID.getTaskId(); Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java Tue Jun 21 20:26:30 2011 @@ -85,6 +85,7 @@ public class TestMRClientService { taskAttemptStatus.progress = 0.5f; taskAttemptStatus.diagnosticInfo = diagnostic2; taskAttemptStatus.stateString = "RUNNING"; + taskAttemptStatus.taskState = TaskAttemptState.RUNNING; taskAttemptStatus.phase = Phase.MAP; taskAttemptStatus.outputSize = 3; // send the status update Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java Tue Jun 21 20:26:30 2011 @@ -191,6 +191,7 @@ public class TestRuntimeEstimators { status.id = attempt.getID(); status.progress = attempt.getProgress(); status.stateString = attempt.getState().name(); + status.taskState = attempt.getState(); SpeculatorEvent event = new SpeculatorEvent(status, clock.getTime()); speculator.handle(event); } Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java Tue Jun 21 20:26:30 2011 @@ -37,6 +37,7 @@ import org.apache.hadoop.mapreduce.v2.ap import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptCompletionEvent; import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptCompletionEventStatus; import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId; +import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptState; import org.apache.hadoop.mapreduce.v2.api.records.TaskId; import org.apache.hadoop.mapreduce.v2.api.records.TaskState; import org.apache.hadoop.mapreduce.v2.api.records.TaskType; @@ -125,6 +126,16 @@ public class TypeConverter { return taskId; } + public static TaskAttemptState toYarn(org.apache.hadoop.mapred.TaskStatus.State state) { + if (state == org.apache.hadoop.mapred.TaskStatus.State.KILLED_UNCLEAN) { + return TaskAttemptState.KILLED; + } + if (state == org.apache.hadoop.mapred.TaskStatus.State.FAILED_UNCLEAN) { + return TaskAttemptState.FAILED; + } + return TaskAttemptState.valueOf(state.toString()); + } + public static Phase toYarn(org.apache.hadoop.mapred.TaskStatus.Phase phase) { switch (phase) { case STARTING: @@ -211,10 +222,11 @@ public class TypeConverter { org.apache.hadoop.mapreduce.Counters counters = new org.apache.hadoop.mapreduce.Counters(); for (CounterGroup yGrp : yCntrs.getAllCounterGroups().values()) { + counters.addGroup(yGrp.getName(), yGrp.getDisplayName()); for (Counter yCntr : yGrp.getAllCounters().values()) { org.apache.hadoop.mapreduce.Counter c = - counters.findCounter(yGrp.getDisplayName(), - yCntr.getDisplayName()); + counters.findCounter(yGrp.getName(), + yCntr.getName()); c.setValue(yCntr.getValue()); } } Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java Tue Jun 21 20:26:30 2011 @@ -90,4 +90,10 @@ public class JHConfig { public static final String HS_CLIENT_THREADS = HS_PREFIX + "client.threads"; public static final int DEFAULT_HS_CLIENT_THREADS = 10; + +//From JTConfig. May need to be moved elsewhere. + public static final String JOBHISTORY_TASKPROGRESS_NUMBER_SPLITS_KEY = + "mapreduce.jobtracker.jobhistory.task.numberprogresssplits"; + + public static int DEFAULT_NUMBER_PROGRESS_SPLITS = 12; } Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java Tue Jun 21 20:26:30 2011 @@ -37,6 +37,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.PathFilter; import org.apache.hadoop.fs.RemoteIterator; import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.mapreduce.JobID; import org.apache.hadoop.mapreduce.TypeConverter; import org.apache.hadoop.mapreduce.v2.MRConstants; import org.apache.hadoop.mapreduce.v2.api.records.JobId; @@ -123,11 +124,29 @@ public class JobHistoryUtils { } }; + /** + * Checks whether the provided path string is a valid job history file. + * @param pathString the path to be checked. + * @return + */ public static boolean isValidJobHistoryFileName(String pathString) { return pathString.endsWith(JOB_HISTORY_FILE_EXTENSION); } /** + * Returns the jobId from a job history file name. + * @param pathString the path string. + * @return the JobId + * @throws IOException if the filename format is invalid. + */ + public static JobID getJobIDFromHistoryFilePath(String pathString) throws IOException { + String [] parts = pathString.split(File.separator); + String fileNamePart = parts[parts.length -1]; + JobIndexInfo jobIndexInfo = FileNameIndexUtils.getIndexInfo(fileNamePart); + return TypeConverter.fromYarn(jobIndexInfo.getJobId()); + } + + /** * Gets a PathFilter which would match configuration files. * @return */ Modified: hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java (original) +++ hadoop/common/branches/MR-279/mapreduce/mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java Tue Jun 21 20:26:30 2011 @@ -480,7 +480,7 @@ public class TaskLog { // Export the pid of taskJvm to env variable JVM_PID. // Currently pid is not used on Windows if (!Shell.WINDOWS) { - mergedCmd.append("export JVM_PID=`echo $$` ; "); + mergedCmd.append("export JVM_PID=$$; "); } if (setup != null) { Modified: hadoop/common/branches/MR-279/mapreduce/yarn/yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/yarn/yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/yarn/yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java (original) +++ hadoop/common/branches/MR-279/mapreduce/yarn/yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java Tue Jun 21 20:26:30 2011 @@ -152,15 +152,15 @@ public abstract class ResourceCalculator // FIXME // No class given, try a os specific class -// try { -// String osName = System.getProperty("os.name"); -// if (osName.startsWith("Linux")) { -// return new LinuxResourceCalculatorPlugin(); -// } -// } catch (SecurityException se) { -// // Failed to get Operating System name. -// return null; -// } + try { + String osName = System.getProperty("os.name"); + if (osName.startsWith("Linux")) { + return new LinuxResourceCalculatorPlugin(); + } + } catch (SecurityException se) { + // Failed to get Operating System name. + return null; + } // Not supported on this system. return null; Modified: hadoop/common/branches/MR-279/mapreduce/yarn/yarn-server/yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/yarn/yarn-server/yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java?rev=1138166&r1=1138165&r2=1138166&view=diff ============================================================================== --- hadoop/common/branches/MR-279/mapreduce/yarn/yarn-server/yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java (original) +++ hadoop/common/branches/MR-279/mapreduce/yarn/yarn-server/yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java Tue Jun 21 20:26:30 2011 @@ -41,6 +41,7 @@ import org.apache.hadoop.fs.LocalDirAllo import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.security.Credentials; +import org.apache.hadoop.util.Shell; import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.api.ApplicationConstants; import org.apache.hadoop.yarn.api.records.ContainerLaunchContext; @@ -287,6 +288,9 @@ public class ContainerLaunch implements sb.env("YARN_HOME", System.getenv("YARN_HOME")); } sb.env(ApplicationConstants.LOCAL_DIR_ENV, StringUtils.join(",", appDirs)); + if (!Shell.WINDOWS) { + sb.env("JVM_PID", "$$"); + } if (environment != null) { for (Map.Entry env : environment.entrySet()) { sb.env(env.getKey().toString(), env.getValue().toString());