Return-Path: Delivered-To: apmail-lucene-hadoop-commits-archive@locus.apache.org Received: (qmail 15581 invoked from network); 23 Mar 2007 21:01:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 23 Mar 2007 21:01:30 -0000 Received: (qmail 72528 invoked by uid 500); 23 Mar 2007 21:01:38 -0000 Delivered-To: apmail-lucene-hadoop-commits-archive@lucene.apache.org Received: (qmail 72509 invoked by uid 500); 23 Mar 2007 21:01:38 -0000 Mailing-List: contact hadoop-commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-commits@lucene.apache.org Received: (qmail 72500 invoked by uid 99); 23 Mar 2007 21:01:37 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Mar 2007 14:01:37 -0700 X-ASF-Spam-Status: No, hits=-99.5 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO eris.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Mar 2007 14:01:29 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 58D3C1A9838; Fri, 23 Mar 2007 14:01:09 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r521900 - in /lucene/hadoop/trunk: CHANGES.txt src/webapps/job/analysejobhistory.jsp Date: Fri, 23 Mar 2007 21:01:09 -0000 To: hadoop-commits@lucene.apache.org From: cutting@apache.org X-Mailer: svnmailer-1.1.0 Message-Id: <20070323210109.58D3C1A9838@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: cutting Date: Fri Mar 23 14:01:08 2007 New Revision: 521900 URL: http://svn.apache.org/viewvc?view=rev&rev=521900 Log: HADOOP-1101. In web ui, split shuffle statisitics from reduce statistics, and add some averages. Contributed by Devaraj. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/webapps/job/analysejobhistory.jsp Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?view=diff&rev=521900&r1=521899&r2=521900 ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Fri Mar 23 14:01:08 2007 @@ -9,6 +9,9 @@ 2. HADOOP-1063. Fix a race condition in MiniDFSCluster test code. (Hairong Kuang via cutting) + 3. HADOOP-1101. In web ui, split shuffle statistics from reduce + statistics, and add some task averages. (Devaraj Das via cutting) + Release 0.12.2 - 2007-23-17 Modified: lucene/hadoop/trunk/src/webapps/job/analysejobhistory.jsp URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/webapps/job/analysejobhistory.jsp?view=diff&rev=521900&r1=521899&r2=521900 ============================================================================== --- lucene/hadoop/trunk/src/webapps/job/analysejobhistory.jsp (original) +++ lucene/hadoop/trunk/src/webapps/job/analysejobhistory.jsp Fri Mar 23 14:01:08 2007 @@ -48,33 +48,67 @@ JobHistory.Task [] mapTasks = new JobHistory.Task[finishedMaps]; JobHistory.Task [] reduceTasks = new JobHistory.Task[finishedReduces]; int mapIndex = 0 , reduceIndex=0; + long avgMapTime = 0; + long avgReduceTime = 0; + long avgShuffleTime = 0; for( JobHistory.Task task : tasks.values() ) { + long avgFinishTime = (task.getLong(Keys.FINISH_TIME) - + task.getLong(Keys.START_TIME)); if( Values.MAP.name().equals(task.get(Keys.TASK_TYPE)) ){ mapTasks[mapIndex++] = task ; - }else{ - reduceTasks[reduceIndex++] = task; + avgMapTime += avgFinishTime; + }else{ + Map attempts = task.getTaskAttempts(); + for (JobHistory.TaskAttempt attempt : attempts.values()) { + if (attempt.get(Keys.TASK_STATUS).equals(Values.SUCCESS.name())) { + reduceTasks[reduceIndex++] = attempt; + avgShuffleTime += (attempt.getLong(Keys.SHUFFLE_FINISHED) - + attempt.getLong(Keys.START_TIME)); + avgReduceTime += (attempt.getLong(Keys.FINISH_TIME) - + attempt.getLong(Keys.SHUFFLE_FINISHED)); + break; + } + } } } - Comparator c = new Comparator(){ + if (finishedMaps > 0) { + avgMapTime /= finishedMaps; + } + if (finishedReduces > 0) { + avgReduceTime /= finishedReduces; + avgShuffleTime /= finishedReduces; + } + Comparator cMap = new Comparator(){ public int compare(JobHistory.Task t1, JobHistory.Task t2){ Long l1 = new Long(t1.getLong(Keys.FINISH_TIME) - t1.getLong(Keys.START_TIME)); Long l2 = new Long(t2.getLong(Keys.FINISH_TIME) - t2.getLong(Keys.START_TIME)) ; return l2.compareTo(l1); } }; - Arrays.sort(mapTasks, c); - Arrays.sort(reduceTasks, c); + Comparator cShuffle = new Comparator(){ + public int compare(JobHistory.Task t1, JobHistory.Task t2){ + Long l1 = new Long(t1.getLong(Keys.SHUFFLE_FINISHED) - + t1.getLong(Keys.START_TIME)); + Long l2 = new Long(t2.getLong(Keys.SHUFFLE_FINISHED) - + t2.getLong(Keys.START_TIME)) ; + return l2.compareTo(l1); + } + }; + Arrays.sort(mapTasks, cMap); + Arrays.sort(reduceTasks, cShuffle); JobHistory.Task minMap = mapTasks[mapTasks.length-1] ; - JobHistory.Task minReduce = reduceTasks[reduceTasks.length-1] ; + JobHistory.Task minShuffle = reduceTasks[reduceTasks.length-1] ; %>

Time taken by best performing Map task <%=minMap.get(Keys.TASKID) %> : <%=StringUtils.formatTimeDiff(minMap.getLong(Keys.FINISH_TIME), minMap.getLong(Keys.START_TIME) ) %>

+

Average time taken by Map tasks: +<%=StringUtils.formatTimeDiff(avgMapTime, 0) %>

Worse performing map tasks

@@ -90,10 +124,96 @@ } %>
Task IdTime taken
+<% + Comparator cFinishMapRed = + new Comparator() { + public int compare(JobHistory.Task t1, JobHistory.Task t2){ + Long l1 = new Long(t1.getLong(Keys.FINISH_TIME)); + Long l2 = new Long(t2.getLong(Keys.FINISH_TIME)); + return l2.compareTo(l1); + } + }; + Arrays.sort(mapTasks, cFinishMapRed); + JobHistory.Task lastMap = mapTasks[0] ; +%> +

The last Map task +<%=lastMap.get(Keys.TASKID) %> +finished at (relative to the Job launch time): +<%=StringUtils.getFormattedTimeWithDiff(dateFormat, + lastMap.getLong(Keys.FINISH_TIME), + job.getLong(Keys.LAUNCH_TIME) ) %>

+
+

Time taken by best performing shuffle +<%=minShuffle.get(Keys.TASKID)%> : +<%=StringUtils.formatTimeDiff(minShuffle.getLong(Keys.SHUFFLE_FINISHED), + minShuffle.getLong(Keys.START_TIME) ) %>

+

Average time taken by Shuffle: +<%=StringUtils.formatTimeDiff(avgShuffleTime, 0) %>

+

Worse performing Shuffle(s)

+ + +<% + for( int i=0;i + + + + +<% + } +%> +
Task IdTime taken
+ <%=reduceTasks[i].get(Keys.TASKID) %><%= + StringUtils.formatTimeDiff( + reduceTasks[i].getLong(Keys.SHUFFLE_FINISHED), + reduceTasks[i].getLong(Keys.START_TIME)) %> +
+<% + Comparator cFinishShuffle = + new Comparator() { + public int compare(JobHistory.Task t1, JobHistory.Task t2){ + Long l1 = new Long(t1.getLong(Keys.SHUFFLE_FINISHED)); + Long l2 = new Long(t2.getLong(Keys.SHUFFLE_FINISHED)); + return l2.compareTo(l1); + } + }; + Arrays.sort(reduceTasks, cFinishShuffle); + JobHistory.Task lastShuffle = reduceTasks[0] ; +%> + +

The last Shuffle +<%=lastShuffle.get(Keys.TASKID)%> + finished at (relative to the Job launch time): +<%=StringUtils.getFormattedTimeWithDiff(dateFormat, + lastShuffle.getLong(Keys.SHUFFLE_FINISHED), + job.getLong(Keys.LAUNCH_TIME) ) %>

+ +<% + Comparator cReduce = new Comparator(){ + public int compare(JobHistory.Task t1, JobHistory.Task t2){ + Long l1 = new Long(t1.getLong(Keys.FINISH_TIME) - + t1.getLong(Keys.SHUFFLE_FINISHED)); + Long l2 = new Long(t2.getLong(Keys.FINISH_TIME) - + t2.getLong(Keys.SHUFFLE_FINISHED)); + return l2.compareTo(l1); + } + }; + Arrays.sort(reduceTasks, cReduce); + JobHistory.Task minReduce = reduceTasks[reduceTasks.length-1] ; +%> +

Time taken by best performing Reduce task : -<%=minReduce.get(Keys.TASKID) %> : <%=StringUtils.formatTimeDiff(minReduce.getLong(Keys.FINISH_TIME), minReduce.getLong(Keys.START_TIME) ) %>

+<%=minReduce.get(Keys.TASKID) %> : +<%=StringUtils.formatTimeDiff(minReduce.getLong(Keys.FINISH_TIME), + minReduce.getLong(Keys.SHUFFLE_FINISHED) ) %> +

Average time taken by Reduce tasks: +<%=StringUtils.formatTimeDiff(avgReduceTime, 0) %>

Worse performing reduce tasks

@@ -103,11 +223,25 @@ - + <% } %>
Task IdTime taken
<%=reduceTasks[i].get(Keys.TASKID) %><%=StringUtils.formatTimeDiff(reduceTasks[i].getLong(Keys.FINISH_TIME), reduceTasks[i].getLong(Keys.START_TIME)) %><%=StringUtils.formatTimeDiff( + reduceTasks[i].getLong(Keys.FINISH_TIME), + reduceTasks[i].getLong(Keys.SHUFFLE_FINISHED)) %>
+<% + Arrays.sort(reduceTasks, cFinishMapRed); + JobHistory.Task lastReduce = reduceTasks[0] ; +%> + +

The last Reduce task +<%=lastReduce.get(Keys.TASKID)%> + finished at (relative to the Job launch time): +<%=StringUtils.getFormattedTimeWithDiff(dateFormat, + lastReduce.getLong(Keys.FINISH_TIME), + job.getLong(Keys.LAUNCH_TIME) ) %>