Return-Path: Delivered-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Received: (qmail 34197 invoked from network); 5 Feb 2011 00:41:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Feb 2011 00:41:24 -0000 Received: (qmail 24165 invoked by uid 500); 5 Feb 2011 00:41:24 -0000 Delivered-To: apmail-hadoop-mapreduce-commits-archive@hadoop.apache.org Received: (qmail 24107 invoked by uid 500); 5 Feb 2011 00:41:23 -0000 Mailing-List: contact mapreduce-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-commits@hadoop.apache.org Received: (qmail 24099 invoked by uid 99); 5 Feb 2011 00:41:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 00:41:23 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 00:41:21 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 96F11238896F; Sat, 5 Feb 2011 00:40:59 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1067361 - in /hadoop/mapreduce/trunk: ./ src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/ src/java/org/apache/hadoop/mapred/ Date: Sat, 05 Feb 2011 00:40:59 -0000 To: mapreduce-commits@hadoop.apache.org From: shv@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110205004059.96F11238896F@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: shv Date: Sat Feb 5 00:40:59 2011 New Revision: 1067361 URL: http://svn.apache.org/viewvc?rev=1067361&view=rev Log: MAPREDUCE-1825. jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps and finishedReduces when job is not initialized. Contributed by Scott Chen. Modified: hadoop/mapreduce/trunk/CHANGES.txt hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairSchedulerServlet.java hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java Modified: hadoop/mapreduce/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/CHANGES.txt?rev=1067361&r1=1067360&r2=1067361&view=diff ============================================================================== --- hadoop/mapreduce/trunk/CHANGES.txt (original) +++ hadoop/mapreduce/trunk/CHANGES.txt Sat Feb 5 00:40:59 2011 @@ -547,6 +547,10 @@ Release 0.22.0 - Unreleased MAPREDUCE-2304. Fix TestMRCLI to allow hostname with a hyphen (-). (Priyo Mustafi via shv) + MAPREDUCE-1825 jobqueue_details.jsp and FairSchedulerServelet should not + call finishedMaps and finishedReduces when job is not initialized. + (Scott Chen via shv) + Release 0.21.1 - Unreleased NEW FEATURES Modified: hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java?rev=1067361&r1=1067360&r2=1067361&view=diff ============================================================================== --- hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java (original) +++ hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java Sat Feb 5 00:40:59 2011 @@ -669,6 +669,9 @@ public class FairScheduler extends TaskS Map userJobs = new HashMap(); Map poolJobs = new HashMap(); for (JobInProgress job: jobs) { + if (!job.inited()) { + continue; + } String user = job.getJobConf().getUser(); String pool = poolMgr.getPoolName(job); int userCount = userJobs.containsKey(user) ? userJobs.get(user) : 0; Modified: hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairSchedulerServlet.java URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairSchedulerServlet.java?rev=1067361&r1=1067360&r2=1067361&view=diff ============================================================================== --- hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairSchedulerServlet.java (original) +++ hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairSchedulerServlet.java Sat Feb 5 00:40:59 2011 @@ -30,6 +30,7 @@ import java.util.Collection; import java.util.Collections; import java.util.Comparator; import java.util.Date; +import java.util.Iterator; import java.util.List; import javax.servlet.ServletContext; @@ -83,7 +84,7 @@ public class FairSchedulerServlet extend boolean advancedView = request.getParameter("advanced") != null; if (JSPUtil.privateActionsAllowed(jobTracker.conf) && request.getParameter("setPool") != null) { - Collection runningJobs = jobTracker.getRunningJobs(); + Collection runningJobs = getInitedJobs(); PoolManager poolMgr = null; synchronized (scheduler) { poolMgr = scheduler.getPoolManager(); @@ -104,7 +105,7 @@ public class FairSchedulerServlet extend } if (JSPUtil.privateActionsAllowed(jobTracker.conf) && request.getParameter("setPriority") != null) { - Collection runningJobs = jobTracker.getRunningJobs(); + Collection runningJobs = getInitedJobs(); JobPriority priority = JobPriority.valueOf(request.getParameter( "setPriority")); String jobId = request.getParameter("jobid"); @@ -246,7 +247,7 @@ public class FairSchedulerServlet extend (advancedView ? "Weight" : "")); out.print("\n"); synchronized (jobTracker) { - Collection runningJobs = jobTracker.getRunningJobs(); + Collection runningJobs = getInitedJobs(); synchronized (scheduler) { for (JobInProgress job: runningJobs) { JobProfile profile = job.getProfile(); @@ -323,4 +324,19 @@ public class FairSchedulerServlet extend html.append("\n"); return html.toString(); } + + /** + * Obtained all initialized jobs + */ + private Collection getInitedJobs() { + Collection runningJobs = jobTracker.getRunningJobs(); + for (Iterator it = runningJobs.iterator(); it.hasNext();) { + JobInProgress job = it.next(); + if (!job.inited()) { + it.remove(); + } + } + return runningJobs; + } + } Modified: hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java?rev=1067361&r1=1067360&r2=1067361&view=diff ============================================================================== --- hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java (original) +++ hadoop/mapreduce/trunk/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java Sat Feb 5 00:40:59 2011 @@ -99,6 +99,9 @@ public class JobSchedulable extends Sche @Override public int getRunningTasks() { + if (!job.inited()) { + return 0; + } return taskType == TaskType.MAP ? job.runningMaps() : job.runningReduces(); } Modified: hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java?rev=1067361&r1=1067360&r2=1067361&view=diff ============================================================================== --- hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java (original) +++ hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java Sat Feb 5 00:40:59 2011 @@ -264,6 +264,15 @@ class JSPUtil { public static String generateJobTable(String label, Collection jobs , int refresh, int rowId, JobConf conf) throws IOException { + // Remove uninitialized jobs because calling JobInProgress synchronized + // methods while job initialization takes long time + for (Iterator it = jobs.iterator(); it.hasNext();) { + JobInProgress job = it.next(); + if (!job.inited()) { + it.remove(); + } + } + boolean isModifiable = label.equals("Running") && privateActionsAllowed(conf); StringBuilder sb = new StringBuilder();