Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Wed, 9 May 2012 10:58:01 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <115459593.43658.1336561081934.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <63160998.1186.1321934680645.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated MAPREDUCE-3451:
-------------------------------

    Attachment: MAPREDUCE-3451.v5.patch

Per MAPREDUCE-3812, I've tweaked your patch a bit to make it use the new configs for min/max RM allocation (memory for containers).

One thing I noticed when tweaking your patch, is that there are quite a bit of trailing white-spaces and possibly a few occurrences of tab+space mix you may wanna clean up.

The diff I've added to your patch overall is below:

{code}
diff --git a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
index 35eafc5..e888be8 100644
--- a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
+++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
@@ -4,6 +4,7 @@ import java.io.File;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources;
 
 public class FairSchedulerConfiguration extends Configuration {
@@ -13,15 +14,7 @@ public class FairSchedulerConfiguration extends Configuration {
   
   protected static final String ALLOCATION_FILE = CONF_PREFIX + "allocation.file";
   protected static final String EVENT_LOG_DIR = "eventlog.dir";
-  
-  /** The minimum size container the scheduler will ever allocate. */
-  protected static final String MINIMUM_MEMORY_MB = CONF_PREFIX + "minimum-allocation-mb";
-  protected static final int    DEFAULT_MINIMUM_MEMORY_MB = 512;
-  
-  /** The maximum size container the scheduler will ever allocate. */
-  protected static final String MAXIMUM_ALLOCATION_MB = CONF_PREFIX + "maximum-allocation-mb";
-  protected static final int    DEFAULT_MAXIMUM_ALLOCATION_MB = 10240;
-  
+
   /** Whether to use the user name as the queue name (instead of "default") if
    * the request does not specify a queue. */
   protected static final String  USER_AS_DEFAULT_QUEUE = CONF_PREFIX + "user-as-default-queue";
@@ -60,17 +53,21 @@ public class FairSchedulerConfiguration extends Configuration {
     super(conf);
     addResource(FS_CONFIGURATION_FILE);
   }
-    
+
   public Resource getMinimumMemoryAllocation() {
-    int mem = getInt(MINIMUM_MEMORY_MB, DEFAULT_MINIMUM_MEMORY_MB);
+    int mem = getInt(
+        YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
+        YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
     return Resources.createResource(mem);
   }
-  
+
   public Resource getMaximumMemoryAllocation() {
-    int mem = getInt(MAXIMUM_ALLOCATION_MB, DEFAULT_MAXIMUM_ALLOCATION_MB);
+    int mem = getInt(
+        YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
+        YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB);
     return Resources.createResource(mem);
   }
-  
+
   public boolean getUserAsDefaultQueue() {
     return getBoolean(USER_AS_DEFAULT_QUEUE, DEFAULT_USER_AS_DEFAULT_QUEUE);
   }
{code}

bq. Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress?

I didn't take a look at which specific mem-var you're discussing here, but what happens if one reads it when its being updated? Does your locking mech cover this scenario too? I remember making the same mistake a while ago at https://issues.apache.org/jira/browse/MAPREDUCE-1347?focusedCommentId=13003257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13003257 but that may be irrelevant here.

In case you wish to add it to ignores, see file {{hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml}} and add appropriate entries for your discovered warnings to suppress them. I'd recommend suppressing them after the review/over another ticket and just leave as a comment for now that this may not be harmful as it looks.
                
> Port Fair Scheduler to MR2
> --------------------------
>
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
>         Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira