Return-Path: X-Original-To: apmail-hadoop-yarn-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 59D1D10856 for ; Tue, 30 Apr 2013 23:03:15 +0000 (UTC) Received: (qmail 13745 invoked by uid 500); 30 Apr 2013 23:03:15 -0000 Delivered-To: apmail-hadoop-yarn-commits-archive@hadoop.apache.org Received: (qmail 13704 invoked by uid 500); 30 Apr 2013 23:03:14 -0000 Mailing-List: contact yarn-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-commits@hadoop.apache.org Delivered-To: mailing list yarn-commits@hadoop.apache.org Received: (qmail 13621 invoked by uid 99); 30 Apr 2013 23:03:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Apr 2013 23:03:14 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Apr 2013 23:03:04 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 059472388B45; Tue, 30 Apr 2013 23:02:42 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1477868 - in /hadoop/common/branches/HDFS-2802/hadoop-yarn-project: ./ hadoop-yarn/ hadoop-yarn/hadoop-yarn-applications/ hadoop-yarn/hadoop-yarn-server/ hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apach... Date: Tue, 30 Apr 2013 23:02:40 -0000 To: yarn-commits@hadoop.apache.org From: szetszwo@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20130430230242.059472388B45@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: szetszwo Date: Tue Apr 30 23:02:35 2013 New Revision: 1477868 URL: http://svn.apache.org/r1477868 Log: Merge r1476453 through r1477867 from trunk. Removed: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/pom.xml hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeHealthScriptRunner.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMExpiry.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestRMNMRPCResponseId.java hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/pom.xml Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt Tue Apr 30 23:02:35 2013 @@ -41,7 +41,7 @@ Trunk - Unreleased classpath with new process's environment variables and localized resources (Chris Nauroth via bikas) - BREAKDOWN OF HADOOP-8562 SUBTASKS + BREAKDOWN OF HADOOP-8562 SUBTASKS AND RELATED JIRAS YARN-158. Yarn creating package-info.java must not depend on sh. (Chris Nauroth via suresh) @@ -70,6 +70,10 @@ Trunk - Unreleased YARN-359. Fixing commands for container signalling in Windows. (Chris Nauroth via vinodkv) + + YARN-506. Move to common utils FileUtil#setReadable/Writable/Executable and + FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) + Release 2.0.5-beta - UNRELEASED @@ -193,6 +197,10 @@ Release 2.0.5-beta - UNRELEASED YARN-591. Moved RM recovery related records out of public API as they do not belong there. (vinodkv) + YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager + to separate out various validation checks depending on whether they rely on + RM configuration or not. (Zhijie Shen via vinodkv) + OPTIMIZATIONS BUG FIXES @@ -312,6 +320,9 @@ Release 2.0.5-beta - UNRELEASED YARN-289. Fair scheduler allows reservations that won't fit on node. (Sandy Ryza via tomwhite) + YARN-576. Modified ResourceManager to reject NodeManagers that don't satisy + minimum resource requirements. (Kenji Kikushima via vinodkv) + Release 2.0.4-alpha - UNRELEASED INCOMPATIBLE CHANGES Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/pom.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/pom.xml?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/pom.xml (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/pom.xml Tue Apr 30 23:02:35 2013 @@ -28,32 +28,15 @@ hadoop-yarn-applications pom + + ${basedir}/../../../../hadoop-common-project/hadoop-common/target + + hadoop-yarn-applications-distributedshell hadoop-yarn-applications-unmanaged-am-launcher - - - - org.apache.maven.plugins - maven-surefire-plugin - - - - ${basedir}/../../../../hadoop-common-project/hadoop-common/target - - - - listener - org.apache.hadoop.test.TimedOutTestsListener - - - - - - - clover Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeHealthScriptRunner.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeHealthScriptRunner.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeHealthScriptRunner.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeHealthScriptRunner.java Tue Apr 30 23:02:35 2013 @@ -28,6 +28,7 @@ import java.util.TimerTask; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.util.Shell.ExitCodeException; import org.apache.hadoop.util.Shell.ShellCommandExecutor; import org.apache.hadoop.util.StringUtils; @@ -321,7 +322,7 @@ public class NodeHealthScriptRunner exte return false; } File f = new File(nodeHealthScript); - return f.exists() && f.canExecute(); + return f.exists() && FileUtil.canExecute(f); } private synchronized void setHealthStatus(boolean isHealthy, String output) { Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java Tue Apr 30 23:02:35 2013 @@ -35,6 +35,7 @@ import java.util.regex.Pattern; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; @@ -307,7 +308,7 @@ public class CgroupsLCEResourcesHandler if (controllerPath != null) { File f = new File(controllerPath + "/" + this.cgroupPrefix); - if (f.canWrite()) { + if (FileUtil.canWrite(f)) { controllerPaths.put(CONTROLLER_CPU, controllerPath); } else { throw new IOException("Not able to enforce cpu weights; cannot write " Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java Tue Apr 30 23:02:35 2013 @@ -36,6 +36,7 @@ import org.apache.commons.logging.LogFac import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileContext; import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.yarn.api.records.ContainerId; @@ -104,9 +105,7 @@ public class TestLinuxContainerExecutor FileContext files = FileContext.getLocalFSFileContext(); Path workSpacePath = new Path(workSpace.getAbsolutePath()); files.mkdir(workSpacePath, null, true); - workSpace.setReadable(true, false); - workSpace.setExecutable(true, false); - workSpace.setWritable(true, false); + FileUtil.chmod(workSpace.getAbsolutePath(), "777"); File localDir = new File(workSpace.getAbsoluteFile(), "localDir"); files.mkdir(new Path(localDir.getAbsolutePath()), new FsPermission("777"), false); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java Tue Apr 30 23:02:35 2013 @@ -37,6 +37,7 @@ import junit.framework.Assert; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.api.records.ContainerId; @@ -78,8 +79,8 @@ public class TestLinuxContainerExecutorW @Before public void setup() { File f = new File("./src/test/resources/mock-container-executor"); - if(!f.canExecute()) { - f.setExecutable(true); + if(!FileUtil.canExecute(f)) { + FileUtil.setExecutable(f, true); } String executorPath = f.getAbsolutePath(); Configuration conf = new Configuration(); @@ -140,8 +141,8 @@ public class TestLinuxContainerExecutorW // set the scheduler priority to make sure still works with nice -n prio File f = new File("./src/test/resources/mock-container-executor"); - if (!f.canExecute()) { - f.setExecutable(true); + if (!FileUtil.canExecute(f)) { + FileUtil.setExecutable(f, true); } String executorPath = f.getAbsolutePath(); Configuration conf = new Configuration(); @@ -204,8 +205,8 @@ public class TestLinuxContainerExecutorW // reinitialize executer File f = new File("./src/test/resources/mock-container-executer-with-error"); - if (!f.canExecute()) { - f.setExecutable(true); + if (!FileUtil.canExecute(f)) { + FileUtil.setExecutable(f, true); } String executorPath = f.getAbsolutePath(); Configuration conf = new Configuration(); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java Tue Apr 30 23:02:35 2013 @@ -28,6 +28,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileContext; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.yarn.api.records.NodeHealthStatus; import org.apache.hadoop.yarn.conf.YarnConfiguration; @@ -81,7 +82,7 @@ public class TestNodeHealthService { pw.println(scriptStr); pw.flush(); pw.close(); - nodeHealthscriptFile.setExecutable(setExecutable); + FileUtil.setExecutable(nodeHealthscriptFile, setExecutable); } @Test Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Tue Apr 30 23:02:35 2013 @@ -34,6 +34,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.UnsupportedFileSystemException; import org.apache.hadoop.util.Shell; @@ -99,7 +100,7 @@ public class TestContainerLaunch extends String timeoutCommand = Shell.WINDOWS ? "@echo \"hello\"" : "echo \"hello\""; PrintWriter writer = new PrintWriter(new FileOutputStream(shellFile)); - shellFile.setExecutable(true); + FileUtil.setExecutable(shellFile, true); writer.println(timeoutCommand); writer.close(); @@ -123,7 +124,7 @@ public class TestContainerLaunch extends ContainerLaunch.writeLaunchEnv(fos, env, resources, commands); fos.flush(); fos.close(); - tempFile.setExecutable(true); + FileUtil.setExecutable(tempFile, true); Shell.ShellCommandExecutor shexc = new Shell.ShellCommandExecutor(new String[]{tempFile.getAbsolutePath()}, tmpDir); @@ -367,7 +368,7 @@ public class TestContainerLaunch extends writer.println("while true; do\nsleep 1s;\ndone"); } writer.close(); - scriptFile.setExecutable(true); + FileUtil.setExecutable(scriptFile, true); ContainerLaunchContext containerLaunchContext = recordFactory.newRecordInstance(ContainerLaunchContext.class); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java Tue Apr 30 23:02:35 2013 @@ -38,7 +38,6 @@ import org.apache.hadoop.security.UserGr import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod; import org.apache.hadoop.security.authorize.PolicyProvider; import org.apache.hadoop.security.token.Token; -import org.apache.hadoop.util.ExitUtil; import org.apache.hadoop.yarn.api.ClientRMProtocol; import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenRequest; import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenResponse; @@ -72,7 +71,6 @@ import org.apache.hadoop.yarn.api.record import org.apache.hadoop.yarn.api.records.NodeReport; import org.apache.hadoop.yarn.api.records.QueueInfo; import org.apache.hadoop.yarn.api.records.Resource; -import org.apache.hadoop.yarn.api.records.ResourceRequest; import org.apache.hadoop.yarn.api.records.YarnClusterMetrics; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.exceptions.YarnRemoteException; @@ -83,15 +81,11 @@ import org.apache.hadoop.yarn.ipc.YarnRP import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier; import org.apache.hadoop.yarn.server.RMDelegationTokenSecretManager; import org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger.AuditConstants; -import org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEvent; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.InvalidResourceRequestException; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNodeReport; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler; import org.apache.hadoop.yarn.server.resourcemanager.security.authorize.RMPolicyProvider; import org.apache.hadoop.yarn.server.security.ApplicationACLsManager; @@ -266,48 +260,61 @@ public class ClientRMService extends Abs ApplicationSubmissionContext submissionContext = request .getApplicationSubmissionContext(); ApplicationId applicationId = submissionContext.getApplicationId(); - String user = submissionContext.getAMContainerSpec().getUser(); + + // ApplicationSubmissionContext needs to be validated for safety - only + // those fields that are independent of the RM's configuration will be + // checked here, those that are dependent on RM configuration are validated + // in RMAppManager. + + String user = null; try { + // Safety user = UserGroupInformation.getCurrentUser().getShortUserName(); - if (rmContext.getRMApps().get(applicationId) != null) { - throw new IOException("Application with id " + applicationId - + " is already present! Cannot add a duplicate!"); - } - - // Safety submissionContext.getAMContainerSpec().setUser(user); + } catch (IOException ie) { + LOG.warn("Unable to get the current user.", ie); + RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST, + ie.getMessage(), "ClientRMService", + "Exception in submitting application", applicationId); + throw RPCUtil.getRemoteException(ie); + } - // Check whether AM resource requirements are within required limits - if (!submissionContext.getUnmanagedAM()) { - ResourceRequest amReq = BuilderUtils.newResourceRequest( - RMAppAttemptImpl.AM_CONTAINER_PRIORITY, ResourceRequest.ANY, - submissionContext.getResource(), 1); - try { - SchedulerUtils.validateResourceRequest(amReq, - scheduler.getMaximumResourceCapability()); - } catch (InvalidResourceRequestException e) { - LOG.warn("RM app submission failed in validating AM resource request" - + " for application " + applicationId, e); - throw RPCUtil.getRemoteException(e); - } - } + // Though duplication will checked again when app is put into rmContext, + // but it is good to fail the invalid submission as early as possible. + if (rmContext.getRMApps().get(applicationId) != null) { + String message = "Application with id " + applicationId + + " is already present! Cannot add a duplicate!"; + LOG.warn(message); + RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST, + message, "ClientRMService", "Exception in submitting application", + applicationId); + throw RPCUtil.getRemoteException(message); + } + + if (submissionContext.getQueue() == null) { + submissionContext.setQueue(YarnConfiguration.DEFAULT_QUEUE_NAME); + } + if (submissionContext.getApplicationName() == null) { + submissionContext.setApplicationName( + YarnConfiguration.DEFAULT_APPLICATION_NAME); + } - // This needs to be synchronous as the client can query - // immediately following the submission to get the application status. - // So call handle directly and do not send an event. - rmAppManager.handle(new RMAppManagerSubmitEvent(submissionContext, System - .currentTimeMillis())); + try { + // call RMAppManager to submit application directly + rmAppManager.submitApplication(submissionContext, + System.currentTimeMillis(), false); LOG.info("Application with id " + applicationId.getId() + " submitted by user " + user); RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST, "ClientRMService", applicationId); - } catch (IOException ie) { - LOG.info("Exception in submitting application", ie); - RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST, - ie.getMessage(), "ClientRMService", + } catch (YarnRemoteException e) { + LOG.info("Exception in submitting application with id " + + applicationId.getId(), e); + RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST, + e.getMessage(), "ClientRMService", "Exception in submitting application", applicationId); - throw RPCUtil.getRemoteException(ie); + throw e; } SubmitApplicationResponse response = recordFactory Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java Tue Apr 30 23:02:35 2013 @@ -31,8 +31,10 @@ import org.apache.hadoop.security.UserGr import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; +import org.apache.hadoop.yarn.api.records.ResourceRequest; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.exceptions.YarnRemoteException; import org.apache.hadoop.yarn.ipc.RPCUtil; import org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger.AuditConstants; import org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore; @@ -45,8 +47,12 @@ import org.apache.hadoop.yarn.server.res import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.InvalidResourceRequestException; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler; import org.apache.hadoop.yarn.server.security.ApplicationACLsManager; +import org.apache.hadoop.yarn.util.BuilderUtils; /** * This class manages the list of applications for the resource manager. @@ -233,64 +239,77 @@ public class RMAppManager implements Eve @SuppressWarnings("unchecked") protected void submitApplication( ApplicationSubmissionContext submissionContext, long submitTime, - boolean isRecovered) { + boolean isRecovered) throws YarnRemoteException { ApplicationId applicationId = submissionContext.getApplicationId(); - RMApp application = null; - try { - // Sanity checks - if (submissionContext.getQueue() == null) { - submissionContext.setQueue(YarnConfiguration.DEFAULT_QUEUE_NAME); - } - if (submissionContext.getApplicationName() == null) { - submissionContext.setApplicationName( - YarnConfiguration.DEFAULT_APPLICATION_NAME); - } - - // Create RMApp - application = - new RMAppImpl(applicationId, rmContext, this.conf, - submissionContext.getApplicationName(), - submissionContext.getAMContainerSpec().getUser(), - submissionContext.getQueue(), - submissionContext, this.scheduler, this.masterService, - submitTime); - - // Sanity check - duplicate? - if (rmContext.getRMApps().putIfAbsent(applicationId, application) != - null) { - String message = "Application with id " + applicationId - + " is already present! Cannot add a duplicate!"; - LOG.info(message); - throw RPCUtil.getRemoteException(message); - } + // Validation of the ApplicationSubmissionContext needs to be completed + // here. Only those fields that are dependent on RM's configuration are + // checked here as they have to be validated whether they are part of new + // submission or just being recovered. + + // Check whether AM resource requirements are within required limits + if (!submissionContext.getUnmanagedAM()) { + ResourceRequest amReq = BuilderUtils.newResourceRequest( + RMAppAttemptImpl.AM_CONTAINER_PRIORITY, ResourceRequest.ANY, + submissionContext.getResource(), 1); + try { + SchedulerUtils.validateResourceRequest(amReq, + scheduler.getMaximumResourceCapability()); + } catch (InvalidResourceRequestException e) { + LOG.warn("RM app submission failed in validating AM resource request" + + " for application " + applicationId, e); + throw RPCUtil.getRemoteException(e); + } + } - // Inform the ACLs Manager - this.applicationACLsManager.addApplication(applicationId, - submissionContext.getAMContainerSpec().getApplicationACLs()); + // Create RMApp + RMApp application = + new RMAppImpl(applicationId, rmContext, this.conf, + submissionContext.getApplicationName(), + submissionContext.getAMContainerSpec().getUser(), + submissionContext.getQueue(), + submissionContext, this.scheduler, this.masterService, + submitTime); + + // Concurrent app submissions with same applicationId will fail here + // Concurrent app submissions with different applicationIds will not + // influence each other + if (rmContext.getRMApps().putIfAbsent(applicationId, application) != + null) { + String message = "Application with id " + applicationId + + " is already present! Cannot add a duplicate!"; + LOG.warn(message); + throw RPCUtil.getRemoteException(message); + } + // Inform the ACLs Manager + this.applicationACLsManager.addApplication(applicationId, + submissionContext.getAMContainerSpec().getApplicationACLs()); + + try { // Setup tokens for renewal if (UserGroupInformation.isSecurityEnabled()) { this.rmContext.getDelegationTokenRenewer().addApplication( applicationId,parseCredentials(submissionContext), submissionContext.getCancelTokensWhenComplete() ); - } - - // All done, start the RMApp - this.rmContext.getDispatcher().getEventHandler().handle( - new RMAppEvent(applicationId, isRecovered ? RMAppEventType.RECOVER: - RMAppEventType.START)); + } } catch (IOException ie) { - LOG.info("RMAppManager submit application exception", ie); - if (application != null) { - // Sending APP_REJECTED is fine, since we assume that the - // RMApp is in NEW state and thus we havne't yet informed the - // Scheduler about the existence of the application - this.rmContext.getDispatcher().getEventHandler().handle( - new RMAppRejectedEvent(applicationId, ie.getMessage())); - } + LOG.warn( + "Unable to add the application to the delegation token renewer.", + ie); + // Sending APP_REJECTED is fine, since we assume that the + // RMApp is in NEW state and thus we havne't yet informed the + // Scheduler about the existence of the application + this.rmContext.getDispatcher().getEventHandler().handle( + new RMAppRejectedEvent(applicationId, ie.getMessage())); + throw RPCUtil.getRemoteException(ie); } + + // All done, start the RMApp + this.rmContext.getDispatcher().getEventHandler().handle( + new RMAppEvent(applicationId, isRecovered ? RMAppEventType.RECOVER: + RMAppEventType.START)); } private Credentials parseCredentials(ApplicationSubmissionContext application) @@ -377,14 +396,6 @@ public class RMAppManager implements Eve checkAppNumCompletedLimit(); } break; - case APP_SUBMIT: - { - ApplicationSubmissionContext submissionContext = - ((RMAppManagerSubmitEvent)event).getSubmissionContext(); - long submitTime = ((RMAppManagerSubmitEvent)event).getSubmitTime(); - submitApplication(submissionContext, submitTime, false); - } - break; default: LOG.error("Invalid eventtype " + event.getType() + ". Ignoring!"); } Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java Tue Apr 30 23:02:35 2013 @@ -19,6 +19,5 @@ package org.apache.hadoop.yarn.server.resourcemanager; public enum RMAppManagerEventType { - APP_SUBMIT, APP_COMPLETED } Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java Tue Apr 30 23:02:35 2013 @@ -78,6 +78,9 @@ public class ResourceTrackerService exte private static final NodeHeartbeatResponse shutDown = recordFactory .newRecordInstance(NodeHeartbeatResponse.class); + private int minAllocMb; + private int minAllocVcores; + static { resync.setNodeAction(NodeAction.RESYNC); @@ -111,6 +114,14 @@ public class ResourceTrackerService exte + YarnConfiguration.RM_NM_HEARTBEAT_INTERVAL_MS + " should be larger than 0."); } + + minAllocMb = conf.getInt( + YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); + minAllocVcores = conf.getInt( + YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES, + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES); + super.init(conf); } @@ -169,6 +180,16 @@ public class ResourceTrackerService exte return response; } + // Check if this node has minimum allocations + if (capability.getMemory() < minAllocMb + || capability.getVirtualCores() < minAllocVcores) { + LOG.info("NodeManager from " + host + + " doesn't satisfy minimum allocations, Sending SHUTDOWN" + + " signal to the NodeManager."); + response.setNodeAction(NodeAction.SHUTDOWN); + return response; + } + if (isSecurityEnabled()) { MasterKey nextMasterKeyForNode = this.containerTokenSecretManager.getCurrentKey(); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java Tue Apr 30 23:02:35 2013 @@ -45,6 +45,7 @@ public class MockNM { private int responseId; private NodeId nodeId; private final int memory; + private final int vCores = 1; private ResourceTrackerService resourceTracker; private final int httpPort = 2; private MasterKey currentMasterKey; @@ -53,9 +54,7 @@ public class MockNM { this.memory = memory; this.resourceTracker = resourceTracker; String[] splits = nodeIdStr.split(":"); - nodeId = Records.newRecord(NodeId.class); - nodeId.setHost(splits[0]); - nodeId.setPort(Integer.parseInt(splits[1])); + nodeId = BuilderUtils.newNodeId(splits[0], Integer.parseInt(splits[1])); } public NodeId getNodeId() { @@ -83,8 +82,7 @@ public class MockNM { RegisterNodeManagerRequest.class); req.setNodeId(nodeId); req.setHttpPort(httpPort); - Resource resource = Records.newRecord(Resource.class); - resource.setMemory(memory); + Resource resource = BuilderUtils.newResource(memory, vCores); req.setResource(resource); RegisterNodeManagerResponse registrationResponse = resourceTracker.registerNodeManager(req); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java Tue Apr 30 23:02:35 2013 @@ -19,6 +19,9 @@ package org.apache.hadoop.yarn.server.resourcemanager; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + import java.util.HashMap; import java.util.List; import java.util.concurrent.ConcurrentMap; @@ -31,12 +34,15 @@ import org.apache.hadoop.yarn.api.record import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; import org.apache.hadoop.yarn.api.records.ContainerLaunchContext; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.event.AsyncDispatcher; import org.apache.hadoop.yarn.event.Dispatcher; import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.exceptions.YarnRemoteException; import org.apache.hadoop.yarn.factories.RecordFactory; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; +import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.MockRMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEvent; @@ -46,11 +52,11 @@ import org.apache.hadoop.yarn.server.res import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler; import org.apache.hadoop.yarn.server.resourcemanager.security.ClientToAMTokenSecretManagerInRM; import org.apache.hadoop.yarn.server.security.ApplicationACLsManager; import org.apache.hadoop.yarn.service.Service; -import org.apache.hadoop.yarn.util.BuilderUtils; +import org.junit.After; +import org.junit.Before; import org.junit.Test; import com.google.common.collect.Lists; @@ -163,9 +169,10 @@ public class TestAppManager{ super.setCompletedAppsMax(max); } public void submitApplication( - ApplicationSubmissionContext submissionContext) { - super.submitApplication( - submissionContext, System.currentTimeMillis(), false); + ApplicationSubmissionContext submissionContext) + throws YarnRemoteException { + super.submitApplication(submissionContext, System.currentTimeMillis(), + false); } } @@ -179,6 +186,40 @@ public class TestAppManager{ } } + private RMContext rmContext; + private TestRMAppManager appMonitor; + private ApplicationSubmissionContext asContext; + private ApplicationId appId; + + @Before + public void setUp() { + long now = System.currentTimeMillis(); + + rmContext = mockRMContext(1, now - 10); + ResourceScheduler scheduler = mockResourceScheduler(); + Configuration conf = new Configuration(); + ApplicationMasterService masterService = + new ApplicationMasterService(rmContext, scheduler); + appMonitor = new TestRMAppManager(rmContext, + new ClientToAMTokenSecretManagerInRM(), scheduler, masterService, + new ApplicationACLsManager(conf), conf); + + appId = MockApps.newAppID(1); + RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); + asContext = + recordFactory.newRecordInstance(ApplicationSubmissionContext.class); + asContext.setApplicationId(appId); + asContext.setAMContainerSpec(mockContainerLaunchContext(recordFactory)); + asContext.setResource(mockResource()); + setupDispatcher(rmContext, conf); + } + + @After + public void tearDown() { + setAppEventType(RMAppEventType.KILL); + ((Service)rmContext.getDispatcher()).stop(); + } + @Test public void testRMAppRetireNone() throws Exception { long now = System.currentTimeMillis(); @@ -334,38 +375,10 @@ public class TestAppManager{ @Test public void testRMAppSubmit() throws Exception { - long now = System.currentTimeMillis(); - - RMContext rmContext = mockRMContext(0, now - 10); - ResourceScheduler scheduler = new CapacityScheduler(); - Configuration conf = new Configuration(); - ApplicationMasterService masterService = - new ApplicationMasterService(rmContext, scheduler); - TestRMAppManager appMonitor = new TestRMAppManager(rmContext, - new ClientToAMTokenSecretManagerInRM(), scheduler, masterService, - new ApplicationACLsManager(conf), conf); - - ApplicationId appID = MockApps.newAppID(1); - RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); - ApplicationSubmissionContext context = - recordFactory.newRecordInstance(ApplicationSubmissionContext.class); - context.setApplicationId(appID); - ContainerLaunchContext amContainer = recordFactory - .newRecordInstance(ContainerLaunchContext.class); - amContainer.setApplicationACLs(new HashMap()); - context.setAMContainerSpec(amContainer); - setupDispatcher(rmContext, conf); - - appMonitor.submitApplication(context); - RMApp app = rmContext.getRMApps().get(appID); + appMonitor.submitApplication(asContext); + RMApp app = rmContext.getRMApps().get(appId); Assert.assertNotNull("app is null", app); - Assert.assertEquals("app id doesn't match", appID, app.getApplicationId()); - Assert.assertEquals("app name doesn't match", - YarnConfiguration.DEFAULT_APPLICATION_NAME, - app.getName()); - Assert.assertEquals("app queue doesn't match", - YarnConfiguration.DEFAULT_QUEUE_NAME, - app.getQueue()); + Assert.assertEquals("app id doesn't match", appId, app.getApplicationId()); Assert.assertEquals("app state doesn't match", RMAppState.NEW, app.getState()); // wait for event to be processed @@ -374,9 +387,8 @@ public class TestAppManager{ timeoutSecs++ < 20) { Thread.sleep(1000); } - Assert.assertEquals("app event type sent is wrong", RMAppEventType.START, getAppEventType()); - setAppEventType(RMAppEventType.KILL); - ((Service)rmContext.getDispatcher()).stop(); + Assert.assertEquals("app event type sent is wrong", RMAppEventType.START, + getAppEventType()); } @Test (timeout = 30000) @@ -390,10 +402,7 @@ public class TestAppManager{ new int[]{ 1, 1, 1, 1 }}; for (int i = 0; i < globalMaxAppAttempts.length; ++i) { for (int j = 0; j < individualMaxAppAttempts.length; ++j) { - long now = System.currentTimeMillis(); - - RMContext rmContext = mockRMContext(0, now - 10); - ResourceScheduler scheduler = new CapacityScheduler(); + ResourceScheduler scheduler = mockResourceScheduler(); Configuration conf = new Configuration(); conf.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, globalMaxAppAttempts[i]); ApplicationMasterService masterService = @@ -402,21 +411,12 @@ public class TestAppManager{ new ClientToAMTokenSecretManagerInRM(), scheduler, masterService, new ApplicationACLsManager(conf), conf); - RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); - ApplicationSubmissionContext context = - recordFactory.newRecordInstance(ApplicationSubmissionContext.class); - ContainerLaunchContext amContainer = recordFactory - .newRecordInstance(ContainerLaunchContext.class); - amContainer.setApplicationACLs(new HashMap()); - context.setAMContainerSpec(amContainer); - setupDispatcher(rmContext, conf); - - ApplicationId appID = MockApps.newAppID(1); - context.setApplicationId(appID); + ApplicationId appID = MockApps.newAppID(i * 4 + j + 1); + asContext.setApplicationId(appID); if (individualMaxAppAttempts[i][j] != 0) { - context.setMaxAppAttempts(individualMaxAppAttempts[i][j]); + asContext.setMaxAppAttempts(individualMaxAppAttempts[i][j]); } - appMonitor.submitApplication(context); + appMonitor.submitApplication(asContext); RMApp app = rmContext.getRMApps().get(appID); Assert.assertEquals("max application attempts doesn't match", expectedNums[i][j], app.getMaxAppAttempts()); @@ -428,96 +428,73 @@ public class TestAppManager{ Thread.sleep(1000); } setAppEventType(RMAppEventType.KILL); - ((Service)rmContext.getDispatcher()).stop(); } } } - @Test (timeout = 3000) - public void testRMAppSubmitWithQueueAndName() throws Exception { - long now = System.currentTimeMillis(); - - RMContext rmContext = mockRMContext(1, now - 10); - ResourceScheduler scheduler = new CapacityScheduler(); - Configuration conf = new Configuration(); - ApplicationMasterService masterService = - new ApplicationMasterService(rmContext, scheduler); - TestRMAppManager appMonitor = new TestRMAppManager(rmContext, - new ClientToAMTokenSecretManagerInRM(), scheduler, masterService, - new ApplicationACLsManager(conf), conf); - - ApplicationId appID = MockApps.newAppID(10); - RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); - ApplicationSubmissionContext context = recordFactory.newRecordInstance(ApplicationSubmissionContext.class); - context.setApplicationId(appID); - context.setApplicationName("testApp1"); - context.setQueue("testQueue"); - ContainerLaunchContext amContainer = recordFactory - .newRecordInstance(ContainerLaunchContext.class); - amContainer - .setApplicationACLs(new HashMap()); - context.setAMContainerSpec(amContainer); + @Test (timeout = 30000) + public void testRMAppSubmitDuplicateApplicationId() throws Exception { + ApplicationId appId = MockApps.newAppID(0); + asContext.setApplicationId(appId); + RMApp appOrig = rmContext.getRMApps().get(appId); + Assert.assertTrue("app name matches but shouldn't", "testApp1" != appOrig.getName()); - setupDispatcher(rmContext, conf); + // our testApp1 should be rejected and original app with same id should be left in place + try { + appMonitor.submitApplication(asContext); + Assert.fail("Exception is expected when applicationId is duplicate."); + } catch (YarnRemoteException e) { + Assert.assertTrue("The thrown exception is not the expectd one.", + e.getMessage().contains("Cannot add a duplicate!")); + } - appMonitor.submitApplication(context); - RMApp app = rmContext.getRMApps().get(appID); + // make sure original app didn't get removed + RMApp app = rmContext.getRMApps().get(appId); Assert.assertNotNull("app is null", app); - Assert.assertEquals("app id doesn't match", appID, app.getApplicationId()); - Assert.assertEquals("app name doesn't match", "testApp1", app.getName()); - Assert.assertEquals("app queue doesn't match", "testQueue", app.getQueue()); - Assert.assertEquals("app state doesn't match", RMAppState.NEW, app.getState()); + Assert.assertEquals("app id doesn't match", appId, app.getApplicationId()); + Assert.assertEquals("app state doesn't match", RMAppState.FINISHED, app.getState()); + } - // wait for event to be processed - int timeoutSecs = 0; - while ((getAppEventType() == RMAppEventType.KILL) && - timeoutSecs++ < 20) { - Thread.sleep(1000); + @Test (timeout = 30000) + public void testRMAppSubmitInvalidResourceRequest() throws Exception { + asContext.setResource(Resources.createResource( + YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB + 1)); + + // submit an app + try { + appMonitor.submitApplication(asContext); + Assert.fail("Application submission should fail because resource" + + " request is invalid."); + } catch (YarnRemoteException e) { + // Exception is expected + Assert.assertTrue("The thrown exception is not" + + " InvalidResourceRequestException", + e.getMessage().startsWith("Invalid resource request")); } - Assert.assertEquals("app event type sent is wrong", RMAppEventType.START, getAppEventType()); - setAppEventType(RMAppEventType.KILL); - ((Service)rmContext.getDispatcher()).stop(); } - @Test - public void testRMAppSubmitError() throws Exception { - long now = System.currentTimeMillis(); - - // specify 1 here and use same appId below so it gets duplicate entry - RMContext rmContext = mockRMContext(1, now - 10); - ResourceScheduler scheduler = new CapacityScheduler(); - Configuration conf = new Configuration(); - ApplicationMasterService masterService = - new ApplicationMasterService(rmContext, scheduler); - TestRMAppManager appMonitor = new TestRMAppManager(rmContext, - new ClientToAMTokenSecretManagerInRM(), scheduler, masterService, - new ApplicationACLsManager(conf), conf); - - ApplicationId appID = MockApps.newAppID(0); - RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); - ApplicationSubmissionContext context = recordFactory.newRecordInstance(ApplicationSubmissionContext.class); - context.setApplicationId(appID); - context.setApplicationName("testApp1"); - context.setQueue("testQueue"); - - setupDispatcher(rmContext, conf); - - RMApp appOrig = rmContext.getRMApps().get(appID); - Assert.assertTrue("app name matches but shouldn't", "testApp1" != appOrig.getName()); - - ContainerLaunchContext clc = - BuilderUtils.newContainerLaunchContext(null, null, null, null, null, - null, null); - context.setAMContainerSpec(clc); - // our testApp1 should be rejected and original app with same id should be left in place - appMonitor.submitApplication(context); - - // make sure original app didn't get removed - RMApp app = rmContext.getRMApps().get(appID); - Assert.assertNotNull("app is null", app); - Assert.assertEquals("app id doesn't match", appID, app.getApplicationId()); - Assert.assertEquals("app name doesn't matches", appOrig.getName(), app.getName()); - ((Service)rmContext.getDispatcher()).stop(); + private static ResourceScheduler mockResourceScheduler() { + ResourceScheduler scheduler = mock(ResourceScheduler.class); + when(scheduler.getMinimumResourceCapability()).thenReturn( + Resources.createResource( + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB)); + when(scheduler.getMaximumResourceCapability()).thenReturn( + Resources.createResource( + YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB)); + return scheduler; + } + + private static ContainerLaunchContext mockContainerLaunchContext( + RecordFactory recordFactory) { + ContainerLaunchContext amContainer = recordFactory.newRecordInstance( + ContainerLaunchContext.class); + amContainer.setApplicationACLs(new HashMap());; + return amContainer; + } + + private static Resource mockResource() { + return Resources.createResource( + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); } } Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java Tue Apr 30 23:02:35 2013 @@ -250,17 +250,70 @@ public class TestClientRMService { rmContext, null, null, null, dtsm); rmService.renewDelegationToken(request); } + + @Test (timeout = 30000) + @SuppressWarnings ("rawtypes") + public void testAppSubmit() throws Exception { + YarnScheduler yarnScheduler = mockYarnScheduler(); + RMContext rmContext = mock(RMContext.class); + mockRMContext(yarnScheduler, rmContext); + RMStateStore stateStore = mock(RMStateStore.class); + when(rmContext.getStateStore()).thenReturn(stateStore); + RMAppManager appManager = new RMAppManager(rmContext, yarnScheduler, + null, mock(ApplicationACLsManager.class), new Configuration()); + when(rmContext.getDispatcher().getEventHandler()).thenReturn( + new EventHandler() { + public void handle(Event event) {} + }); + ClientRMService rmService = + new ClientRMService(rmContext, yarnScheduler, appManager, null, null); + + // without name and queue + ApplicationId appId1 = getApplicationId(100); + SubmitApplicationRequest submitRequest1 = mockSubmitAppRequest( + appId1, null, null); + try { + rmService.submitApplication(submitRequest1); + } catch (YarnRemoteException e) { + Assert.fail("Exception is not expected."); + } + RMApp app1 = rmContext.getRMApps().get(appId1); + Assert.assertNotNull("app doesn't exist", app1); + Assert.assertEquals("app name doesn't match", + YarnConfiguration.DEFAULT_APPLICATION_NAME, app1.getName()); + Assert.assertEquals("app queue doesn't match", + YarnConfiguration.DEFAULT_QUEUE_NAME, app1.getQueue()); + + // with name and queue + String name = MockApps.newAppName(); + String queue = MockApps.newQueue(); + ApplicationId appId2 = getApplicationId(101); + SubmitApplicationRequest submitRequest2 = mockSubmitAppRequest( + appId2, name, queue); + try { + rmService.submitApplication(submitRequest2); + } catch (YarnRemoteException e) { + Assert.fail("Exception is not expected."); + } + RMApp app2 = rmContext.getRMApps().get(appId2); + Assert.assertNotNull("app doesn't exist", app2); + Assert.assertEquals("app name doesn't match", name, app2.getName()); + Assert.assertEquals("app queue doesn't match", queue, app2.getQueue()); + + // duplicate appId + try { + rmService.submitApplication(submitRequest2); + Assert.fail("Exception is expected."); + } catch (YarnRemoteException e) { + Assert.assertTrue("The thrown exception is not expected.", + e.getMessage().contains("Cannot add a duplicate!")); + } + } @Test(timeout=4000) public void testConcurrentAppSubmit() throws IOException, InterruptedException, BrokenBarrierException { - YarnScheduler yarnScheduler = mock(YarnScheduler.class); - when(yarnScheduler.getMinimumResourceCapability()).thenReturn( - Resources.createResource( - YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB)); - when(yarnScheduler.getMaximumResourceCapability()).thenReturn( - Resources.createResource( - YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB)); + YarnScheduler yarnScheduler = mockYarnScheduler(); RMContext rmContext = mock(RMContext.class); mockRMContext(yarnScheduler, rmContext); RMStateStore stateStore = mock(RMStateStore.class); @@ -270,8 +323,10 @@ public class TestClientRMService { final ApplicationId appId1 = getApplicationId(100); final ApplicationId appId2 = getApplicationId(101); - final SubmitApplicationRequest submitRequest1 = mockSubmitAppRequest(appId1); - final SubmitApplicationRequest submitRequest2 = mockSubmitAppRequest(appId2); + final SubmitApplicationRequest submitRequest1 = mockSubmitAppRequest( + appId1, null, null); + final SubmitApplicationRequest submitRequest2 = mockSubmitAppRequest( + appId2, null, null); final CyclicBarrier startBarrier = new CyclicBarrier(2); final CyclicBarrier endBarrier = new CyclicBarrier(2); @@ -319,61 +374,23 @@ public class TestClientRMService { t.join(); } - @Test (timeout = 30000) - public void testInvalidResourceRequestWhenSubmittingApplication() - throws IOException, InterruptedException, BrokenBarrierException { - YarnScheduler yarnScheduler = mock(YarnScheduler.class); - when(yarnScheduler.getMinimumResourceCapability()).thenReturn( - Resources.createResource( - YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB)); - when(yarnScheduler.getMaximumResourceCapability()).thenReturn( - Resources.createResource( - YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB)); - RMContext rmContext = mock(RMContext.class); - mockRMContext(yarnScheduler, rmContext); - RMStateStore stateStore = mock(RMStateStore.class); - when(rmContext.getStateStore()).thenReturn(stateStore); - RMAppManager appManager = new RMAppManager(rmContext, yarnScheduler, - null, mock(ApplicationACLsManager.class), new Configuration()); - - final ApplicationId appId = getApplicationId(100); - final SubmitApplicationRequest submitRequest = mockSubmitAppRequest(appId); - Resource resource = Resources.createResource( - YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB + 1); - when(submitRequest.getApplicationSubmissionContext() - .getResource()).thenReturn(resource); - - final ClientRMService rmService = - new ClientRMService(rmContext, yarnScheduler, appManager, null, null); - - // submit an app - try { - rmService.submitApplication(submitRequest); - Assert.fail("Application submission should fail because resource" + - " request is invalid."); - } catch (YarnRemoteException e) { - // Exception is expected - Assert.assertTrue("The thrown exception is not" + - " InvalidResourceRequestException", - e.getMessage().startsWith("Invalid resource request")); - } - } - - private SubmitApplicationRequest mockSubmitAppRequest(ApplicationId appId) { + private SubmitApplicationRequest mockSubmitAppRequest(ApplicationId appId, + String name, String queue) { String user = MockApps.newUserName(); - String queue = MockApps.newQueue(); ContainerLaunchContext amContainerSpec = mock(ContainerLaunchContext.class); Resource resource = Resources.createResource( YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); - ApplicationSubmissionContext submissionContext = mock(ApplicationSubmissionContext.class); - when(submissionContext.getAMContainerSpec()).thenReturn(amContainerSpec); - when(submissionContext.getAMContainerSpec().getUser()).thenReturn(user); - when(submissionContext.getQueue()).thenReturn(queue); - when(submissionContext.getApplicationId()).thenReturn(appId); - when(submissionContext.getResource()).thenReturn(resource); + ApplicationSubmissionContext submissionContext = + recordFactory.newRecordInstance(ApplicationSubmissionContext.class); + submissionContext.setAMContainerSpec(amContainerSpec); + submissionContext.getAMContainerSpec().setUser(user); + submissionContext.setApplicationName(name); + submissionContext.setQueue(queue); + submissionContext.setApplicationId(appId); + submissionContext.setResource(resource); SubmitApplicationRequest submitRequest = recordFactory.newRecordInstance(SubmitApplicationRequest.class); @@ -429,4 +446,15 @@ public class TestClientRMService { queueName, asContext, yarnScheduler, null , System .currentTimeMillis()); } + + private static YarnScheduler mockYarnScheduler() { + YarnScheduler yarnScheduler = mock(YarnScheduler.class); + when(yarnScheduler.getMinimumResourceCapability()).thenReturn( + Resources.createResource( + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB)); + when(yarnScheduler.getMaximumResourceCapability()).thenReturn( + Resources.createResource( + YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB)); + return yarnScheduler; + } } Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java Tue Apr 30 23:02:35 2013 @@ -32,6 +32,7 @@ import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ContainerStatus; import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.event.Dispatcher; import org.apache.hadoop.yarn.event.DrainDispatcher; @@ -43,6 +44,7 @@ import org.apache.hadoop.yarn.server.api import org.apache.hadoop.yarn.server.api.records.NodeAction; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEvent; +import org.apache.hadoop.yarn.util.BuilderUtils; import org.apache.hadoop.yarn.util.Records; import org.junit.After; import org.junit.Test; @@ -284,6 +286,49 @@ public class TestResourceTrackerService } @Test + public void testNodeRegistrationWithMinimumAllocations() throws Exception { + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, "2048"); + conf.set(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES, "4"); + rm = new MockRM(conf); + rm.start(); + + ResourceTrackerService resourceTrackerService + = rm.getResourceTrackerService(); + RegisterNodeManagerRequest req = Records.newRecord( + RegisterNodeManagerRequest.class); + NodeId nodeId = BuilderUtils.newNodeId("host", 1234); + req.setNodeId(nodeId); + + Resource capability = BuilderUtils.newResource(1024, 1); + req.setResource(capability); + RegisterNodeManagerResponse response1 = + resourceTrackerService.registerNodeManager(req); + Assert.assertEquals(NodeAction.SHUTDOWN,response1.getNodeAction()); + + capability.setMemory(2048); + capability.setVirtualCores(1); + req.setResource(capability); + RegisterNodeManagerResponse response2 = + resourceTrackerService.registerNodeManager(req); + Assert.assertEquals(NodeAction.SHUTDOWN,response2.getNodeAction()); + + capability.setMemory(1024); + capability.setVirtualCores(4); + req.setResource(capability); + RegisterNodeManagerResponse response3 = + resourceTrackerService.registerNodeManager(req); + Assert.assertEquals(NodeAction.SHUTDOWN,response3.getNodeAction()); + + capability.setMemory(2048); + capability.setVirtualCores(4); + req.setResource(capability); + RegisterNodeManagerResponse response4 = + resourceTrackerService.registerNodeManager(req); + Assert.assertEquals(NodeAction.NORMAL,response4.getNodeAction()); + } + + @Test public void testReboot() throws Exception { Configuration conf = new Configuration(); rm = new MockRM(conf); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMExpiry.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMExpiry.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMExpiry.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMExpiry.java Tue Apr 30 23:02:35 2013 @@ -43,6 +43,7 @@ import org.apache.hadoop.yarn.server.res import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeEventType; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType; import org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager; +import org.apache.hadoop.yarn.util.BuilderUtils; import org.apache.hadoop.yarn.util.Records; import org.junit.Before; import org.junit.Test; @@ -127,7 +128,7 @@ public class TestNMExpiry { String hostname1 = "localhost1"; String hostname2 = "localhost2"; String hostname3 = "localhost3"; - Resource capability = recordFactory.newRecordInstance(Resource.class); + Resource capability = BuilderUtils.newResource(1024, 1); RegisterNodeManagerRequest request1 = recordFactory .newRecordInstance(RegisterNodeManagerRequest.class); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestRMNMRPCResponseId.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestRMNMRPCResponseId.java?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestRMNMRPCResponseId.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestRMNMRPCResponseId.java Tue Apr 30 23:02:35 2013 @@ -43,6 +43,7 @@ import org.apache.hadoop.yarn.server.res import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeEventType; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType; import org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager; +import org.apache.hadoop.yarn.util.BuilderUtils; import org.apache.hadoop.yarn.util.Records; import org.junit.After; import org.junit.Assert; @@ -89,7 +90,7 @@ public class TestRMNMRPCResponseId { @Test public void testRPCResponseId() throws IOException { String node = "localhost"; - Resource capability = recordFactory.newRecordInstance(Resource.class); + Resource capability = BuilderUtils.newResource(1024, 1); RegisterNodeManagerRequest request = recordFactory.newRecordInstance(RegisterNodeManagerRequest.class); nodeId = Records.newRecord(NodeId.class); nodeId.setHost(node); Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml Tue Apr 30 23:02:35 2013 @@ -28,26 +28,9 @@ hadoop-yarn-server pom - - - - org.apache.maven.plugins - maven-surefire-plugin - - - - ${basedir}/../../../../hadoop-common-project/hadoop-common/target - - - - listener - org.apache.hadoop.test.TimedOutTestsListener - - - - - - + + ${basedir}/../../../../hadoop-common-project/hadoop-common/target + Modified: hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/pom.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/pom.xml?rev=1477868&r1=1477867&r2=1477868&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/pom.xml (original) +++ hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/pom.xml Tue Apr 30 23:02:35 2013 @@ -30,6 +30,7 @@ true 600000 ${basedir} + ${basedir}/../../../hadoop-common-project/hadoop-common/target @@ -159,10 +160,6 @@ org.apache.maven.plugins maven-surefire-plugin - - - ${basedir}/../../../hadoop-common-project/hadoop-common/target - listener