hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Date Wed, 09 May 2012 10:58:01 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Harsh J updated MAPREDUCE-3451:
-------------------------------

    Attachment: MAPREDUCE-3451.v5.patch

Per MAPREDUCE-3812, I've tweaked your patch a bit to make it use the new configs for min/max
RM allocation (memory for containers).

One thing I noticed when tweaking your patch, is that there are quite a bit of trailing white-spaces
and possibly a few occurrences of tab+space mix you may wanna clean up.

The diff I've added to your patch overall is below:

{code}
diff --git a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
index 35eafc5..e888be8 100644
--- a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
+++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
@@ -4,6 +4,7 @@ import java.io.File;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources;
 
 public class FairSchedulerConfiguration extends Configuration {
@@ -13,15 +14,7 @@ public class FairSchedulerConfiguration extends Configuration {
   
   protected static final String ALLOCATION_FILE = CONF_PREFIX + "allocation.file";
   protected static final String EVENT_LOG_DIR = "eventlog.dir";
-  
-  /** The minimum size container the scheduler will ever allocate. */
-  protected static final String MINIMUM_MEMORY_MB = CONF_PREFIX + "minimum-allocation-mb";
-  protected static final int    DEFAULT_MINIMUM_MEMORY_MB = 512;
-  
-  /** The maximum size container the scheduler will ever allocate. */
-  protected static final String MAXIMUM_ALLOCATION_MB = CONF_PREFIX + "maximum-allocation-mb";
-  protected static final int    DEFAULT_MAXIMUM_ALLOCATION_MB = 10240;
-  
+
   /** Whether to use the user name as the queue name (instead of "default") if
    * the request does not specify a queue. */
   protected static final String  USER_AS_DEFAULT_QUEUE = CONF_PREFIX + "user-as-default-queue";
@@ -60,17 +53,21 @@ public class FairSchedulerConfiguration extends Configuration {
     super(conf);
     addResource(FS_CONFIGURATION_FILE);
   }
-    
+
   public Resource getMinimumMemoryAllocation() {
-    int mem = getInt(MINIMUM_MEMORY_MB, DEFAULT_MINIMUM_MEMORY_MB);
+    int mem = getInt(
+        YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
+        YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
     return Resources.createResource(mem);
   }
-  
+
   public Resource getMaximumMemoryAllocation() {
-    int mem = getInt(MAXIMUM_ALLOCATION_MB, DEFAULT_MAXIMUM_ALLOCATION_MB);
+    int mem = getInt(
+        YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
+        YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB);
     return Resources.createResource(mem);
   }
-  
+
   public boolean getUserAsDefaultQueue() {
     return getBoolean(USER_AS_DEFAULT_QUEUE, DEFAULT_USER_AS_DEFAULT_QUEUE);
   }
{code}

bq. Okay looks like that patch fixed everything except for the findbugs, there are some member
variables which are written only while holding a lock but can be read outside of the lock
and findbugs doesn't like this. I consider these false positives - any idea how to suppress?

I didn't take a look at which specific mem-var you're discussing here, but what happens if
one reads it when its being updated? Does your locking mech cover this scenario too? I remember
making the same mistake a while ago at https://issues.apache.org/jira/browse/MAPREDUCE-1347?focusedCommentId=13003257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13003257
but that may be irrelevant here.

In case you wish to add it to ignores, see file {{hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml}}
and add appropriate entries for your discovered warnings to suppress them. I'd recommend suppressing
them after the review/over another ticket and just leave as a comment for now that this may
not be harmful as it looks.
                
> Port Fair Scheduler to MR2
> --------------------------
>
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
>         Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt,
MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to
MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include
design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message