Return-Path: X-Original-To: apmail-hadoop-yarn-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A320E107DE for ; Sat, 8 Feb 2014 19:06:12 +0000 (UTC) Received: (qmail 69260 invoked by uid 500); 8 Feb 2014 19:06:11 -0000 Delivered-To: apmail-hadoop-yarn-commits-archive@hadoop.apache.org Received: (qmail 69227 invoked by uid 500); 8 Feb 2014 19:06:11 -0000 Mailing-List: contact yarn-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-commits@hadoop.apache.org Delivered-To: mailing list yarn-commits@hadoop.apache.org Received: (qmail 69219 invoked by uid 99); 8 Feb 2014 19:06:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Feb 2014 19:06:11 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Feb 2014 19:06:00 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 0E5612388B34; Sat, 8 Feb 2014 19:05:22 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1566100 [1/3] - in /hadoop/common/branches/HDFS-4685/hadoop-yarn-project: ./ hadoop-yarn/bin/ hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/ hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop... Date: Sat, 08 Feb 2014 19:05:19 -0000 To: yarn-commits@hadoop.apache.org From: cnauroth@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140208190522.0E5612388B34@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: cnauroth Date: Sat Feb 8 19:05:12 2014 New Revision: 1566100 URL: http://svn.apache.org/r1566100 Log: Merge trunk to HDFS-4685. Added: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/ApplicationsRequestScope.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/ApplicationsRequestScope.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ - copied from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEntities.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEntities.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEntity.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEntity.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEvent.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEvent.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEvents.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSEvents.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSPutErrors.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/ATSPutErrors.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/package-info.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/apptimeline/package-info.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProvider.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProvider.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProviderFactory.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProviderFactory.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/FileSystemBasedConfigurationProvider.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/FileSystemBasedConfigurationProvider.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/LocalConfigurationProvider.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/LocalConfigurationProvider.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/ - copied from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/ - copied from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ - copied from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineReader.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineReader.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineStore.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineStore.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineWriter.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineWriter.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/EntityIdentifier.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/EntityIdentifier.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/GenericObjectMapper.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/GenericObjectMapper.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/LeveldbApplicationTimelineStore.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/LeveldbApplicationTimelineStore.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/MemoryApplicationTimelineStore.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/MemoryApplicationTimelineStore.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/NameValuePair.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/NameValuePair.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/package-info.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/package-info.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ - copied from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineStoreTestUtils.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/ApplicationTimelineStoreTestUtils.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/TestGenericObjectMapper.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/TestGenericObjectMapper.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/TestLeveldbApplicationTimelineStore.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/TestLeveldbApplicationTimelineStore.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/TestMemoryApplicationTimelineStore.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/apptimeline/TestMemoryApplicationTimelineStore.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestATSWebServices.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestATSWebServices.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMoveEvent.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMoveEvent.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/core-site.xml - copied unchanged from r1566041, hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/core-site.xml Removed: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestCompositeService.java Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/CHANGES.txt hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/ContainerLaunchFailAppMaster.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestHAUtil.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebApp.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/loghandler/NonAggregatingLogHandler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/loghandler/TestNonAggregatingLogHandler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYARNClusterForHA.java hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/CHANGES.txt?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/CHANGES.txt (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/CHANGES.txt Sat Feb 8 19:05:12 2014 @@ -12,6 +12,14 @@ Trunk - Unreleased YARN-1498. Common scheduler changes for moving apps between queues (Sandy Ryza) + YARN-1504. RM changes for moving apps between queues (Sandy Ryza) + + YARN-1499. Fair Scheduler changes for moving apps between queues (Sandy + Ryza) + + YARN-1497. Command line additions for moving apps between queues (Sandy + Ryza) + IMPROVEMENTS OPTIMIZATIONS @@ -77,6 +85,40 @@ Release 2.4.0 - UNRELEASED YARN-1413. Implemented serving of aggregated-logs in the ApplicationHistory server. (Mayank Bansal via vinodkv) + YARN-1633. Defined user-facing entity, entity-info and event objects related + to Application Timeline feature. (Zhijie Shen via vinodkv) + + YARN-1611. Introduced the concept of a configuration provider which can be + used by ResourceManager to read configuration locally or from remote systems + so as to help RM failover. (Xuan Gong via vinodkv) + + YARN-1659. Defined the ApplicationTimelineStore store as an abstraction for + implementing different storage impls for storing timeline information. + (Billie Rinaldi via vinodkv) + + YARN-1634. Added a testable in-memory implementation of + ApplicationTimelineStore. (Zhijie Shen via vinodkv) + + YARN-1461. Added tags for YARN applications and changed RM to handle them. + (Karthik Kambatla via zjshen) + + YARN-1636. Augmented Application-history server's web-services to also expose + new APIs for retrieving and storing timeline information. (Zhijie Shen via + vinodkv) + + YARN-1490. Introduced the ability to make ResourceManager optionally not kill + all containers when an ApplicationMaster exits. (Jian He via vinodkv) + + YARN-1041. Added the ApplicationMasterProtocol API for applications to use the + ability in ResourceManager to optionally not kill containers when the + ApplicationMaster exits. (Jian He via vinodkv) + + YARN-1566. Changed Distributed Shell to retain containers across application + attempts. (Jian He via vinodkv) + + YARN-1635. Implemented a Leveldb based ApplicationTimelineStore. (Billie + Rinaldi via zjshen) + IMPROVEMENTS YARN-1007. Enhance History Reader interface for Containers. (Mayank Bansal via @@ -94,6 +136,36 @@ Release 2.4.0 - UNRELEASED YARN-1617. Remove ancient comment and surround LOG.debug in AppSchedulingInfo.allocate (Sandy Ryza) + YARN-1639. Modified RM HA configuration handling to have a way of not + requiring separate configuration files for each RM. (Xuan Gong via vinodkv) + + YARN-1668. Modified RM HA handling of admin-acls to be available across RM + failover by making using of a remote configuration-provider. (Xuan Gong via + vinodkv) + + YARN-1667. Modified RM HA handling of super users (with proxying ability) to + be available across RM failover by making using of a remote + configuration-provider. (Xuan Gong via vinodkv) + + YARN-1285. Changed the default value of yarn.acl.enable in yarn-default.xml + to be consistent with what exists (false) in the code and documentation. + (Kenji Kikushima via vinodkv) + + YARN-1669. Modified RM HA handling of protocol level service-ACLS to + be available across RM failover by making using of a remote + configuration-provider. (Xuan Gong via vinodkv) + + YARN-1665. Simplify the configuration of RM HA by having better default + values. (Xuan Gong via vinodkv) + + YARN-1660. Simplified the RM HA configuration to accept and be able to simply + depend just on configuration properties of the form + yarn.resourcemanager.hostname.RMID and use the default ports for all service + addresses. (Xuan Gong via vinodkv) + + YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize + app-attempts separately from apps. (Jian He via vinodkv) + OPTIMIZATIONS BUG FIXES @@ -143,6 +215,24 @@ Release 2.4.0 - UNRELEASED YARN-1632. TestApplicationMasterServices should be under org.apache.hadoop.yarn.server.resourcemanager package (Chen He via jeagles) + YARN-1673. Fix option parsing in YARN's application CLI after it is broken + by YARN-967. (Mayank Bansal via vinodkv) + + YARN-1684. Fixed history server heap size in yarn script. (Billie Rinaldi + via zjshen) + + YARN-1166. Fixed app-specific and attempt-specific QueueMetrics to be + triggered by accordingly app event and attempt event. + + YARN-1689. Made RMAppAttempt get killed when RMApp is at ACCEPTED. (Vinod + Kumar Vavilapalli via zjshen) + + YARN-1661. Fixed DS ApplicationMaster to write the correct exit log. (Vinod + Kumar Vavilapalli via zjshen) + + YARN-1672. YarnConfiguration is missing a default for + yarn.nodemanager.log.retain-seconds (Naren Koneru via kasha) + Release 2.3.0 - UNRELEASED INCOMPATIBLE CHANGES @@ -186,18 +276,8 @@ Release 2.3.0 - UNRELEASED YARN-1029. Added embedded leader election in the ResourceManager. (Karthik Kambatla via vinodkv) - YARN-1490. Introduced the ability to make ResourceManager optionally not kill - all containers when an ApplicationMaster exits. (Jian He via vinodkv) - YARN-1033. Expose RM active/standby state to Web UI and REST API (kasha) - YARN-1041. Added the ApplicationMasterProtocol API for applications to use the - ability in ResourceManager to optionally not kill containers when the - ApplicationMaster exits. (Jian He via vinodkv) - - YARN-1566. Changed Distributed Shell to retain containers across application - attempts. (Jian He via vinodkv) - IMPROVEMENTS YARN-305. Fair scheduler logs too many "Node offered to app" messages. @@ -361,9 +441,6 @@ Release 2.3.0 - UNRELEASED YARN-1541. Changed ResourceManager to invalidate ApplicationMaster host/port information once an AM crashes. (Jian He via vinodkv) - YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize - app-attempts separately from apps. (Jian He via vinodkv) - YARN-1482. Modified WebApplicationProxy to make it work across ResourceManager fail-over. (Xuan Gong via vinodkv) @@ -549,9 +626,6 @@ Release 2.3.0 - UNRELEASED YARN-1574. RMDispatcher should be reset on transition to standby. (Xuan Gong via kasha) - YARN-1166. Fixed app-specific and attempt-specific QueueMetrics to be - triggered by accordingly app event and attempt event. - YARN-1598. HA-related rmadmin commands don't work on a secure cluster (kasha) YARN-1603. Remove two *.orig files which were unexpectedly committed. @@ -576,6 +650,9 @@ Release 2.3.0 - UNRELEASED YARN-1629. IndexOutOfBoundsException in MaxRunningAppsEnforcer (Sandy Ryza) + YARN-1628. Fixed the test failure in TestContainerManagerSecurity. (Vinod + Kumar Vavilapalli via zjshen) + Release 2.2.0 - 2013-10-13 INCOMPATIBLE CHANGES Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn Sat Feb 8 19:05:12 2014 @@ -204,7 +204,7 @@ elif [ "$COMMAND" = "historyserver" ] ; CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/ahs-config/log4j.properties CLASS='org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer' YARN_OPTS="$YARN_OPTS $YARN_HISTORYSERVER_OPTS" - if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then + if [ "$YARN_HISTORYSERVER_HEAPSIZE" != "" ]; then JAVA_HEAP_MAX="-Xmx""$YARN_HISTORYSERVER_HEAPSIZE""m" fi elif [ "$COMMAND" = "nodemanager" ] ; then Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd Sat Feb 8 19:05:12 2014 @@ -207,7 +207,7 @@ goto :eof set CLASSPATH=%CLASSPATH%;%YARN_CONF_DIR%\ahs-config\log4j.properties set CLASS=org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer set YARN_OPTS=%YARN_OPTS% %HADOOP_HISTORYSERVER_OPTS% - if defined YARN_RESOURCEMANAGER_HEAPSIZE ( + if defined YARN_HISTORYSERVER_HEAPSIZE ( set JAVA_HEAP_MAX=-Xmx%YARN_HISTORYSERVER_HEAPSIZE%m ) goto :eof Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java Sat Feb 8 19:05:12 2014 @@ -21,7 +21,6 @@ package org.apache.hadoop.yarn.api.proto import java.util.EnumSet; import java.util.Set; -import org.apache.commons.collections.buffer.UnboundedFifoBuffer; import org.apache.commons.lang.math.LongRange; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceAudience.Public; @@ -56,6 +55,86 @@ public abstract class GetApplicationsReq * ResourceManager. *

* + * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest) + * + *

Setting any of the parameters to null, would just disable that + * filter

+ * + * @param scope {@link ApplicationsRequestScope} to filter by + * @param users list of users to filter by + * @param queues list of scheduler queues to filter by + * @param applicationTypes types of applications + * @param applicationTags application tags to filter by + * @param applicationStates application states to filter by + * @param startRange range of application start times to filter by + * @param finishRange range of application finish times to filter by + * @param limit number of applications to limit to + * @return {@link GetApplicationsRequest} to be used with + * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)} + */ + @Public + @Stable + public static GetApplicationsRequest newInstance( + ApplicationsRequestScope scope, + Set users, + Set queues, + Set applicationTypes, + Set applicationTags, + EnumSet applicationStates, + LongRange startRange, + LongRange finishRange, + Long limit) { + GetApplicationsRequest request = + Records.newRecord(GetApplicationsRequest.class); + if (scope != null) { + request.setScope(scope); + } + request.setUsers(users); + request.setQueues(queues); + request.setApplicationTypes(applicationTypes); + request.setApplicationTags(applicationTags); + request.setApplicationStates(applicationStates); + if (startRange != null) { + request.setStartRange( + startRange.getMinimumLong(), startRange.getMaximumLong()); + } + if (finishRange != null) { + request.setFinishRange( + finishRange.getMinimumLong(), finishRange.getMaximumLong()); + } + if (limit != null) { + request.setLimit(limit); + } + return request; + } + + /** + *

+ * The request from clients to get a report of Applications matching the + * giving application types in the cluster from the + * ResourceManager. + *

+ * + * @param scope {@link ApplicationsRequestScope} to filter by + * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest) + */ + @Public + @Stable + public static GetApplicationsRequest newInstance( + ApplicationsRequestScope scope) { + GetApplicationsRequest request = + Records.newRecord(GetApplicationsRequest.class); + request.setScope(scope); + return request; + } + + /** + *

+ * The request from clients to get a report of Applications matching the + * giving application types in the cluster from the + * ResourceManager. + *

+ * * * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest) */ @@ -257,4 +336,40 @@ public abstract class GetApplicationsReq @Private @Unstable public abstract void setFinishRange(long begin, long end); + + /** + * Get the tags to filter applications on + * + * @return list of tags to filter on + */ + @Private + @Unstable + public abstract Set getApplicationTags(); + + /** + * Set the list of tags to filter applications on + * + * @param tags list of tags to filter on + */ + @Private + @Unstable + public abstract void setApplicationTags(Set tags); + + /** + * Get the {@link ApplicationsRequestScope} of applications to be filtered. + * + * @return {@link ApplicationsRequestScope} of applications to return. + */ + @Private + @Unstable + public abstract ApplicationsRequestScope getScope(); + + /** + * Set the {@link ApplicationsRequestScope} of applications to filter. + * + * @param scope scope to use for filtering applications + */ + @Private + @Unstable + public abstract void setScope(ApplicationsRequestScope scope); } Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java Sat Feb 8 19:05:12 2014 @@ -25,6 +25,8 @@ import org.apache.hadoop.classification. import org.apache.hadoop.yarn.api.ApplicationClientProtocol; import org.apache.hadoop.yarn.util.Records; +import java.util.Set; + /** *

ApplicationReport is a report of an application.

* @@ -321,6 +323,18 @@ public abstract class ApplicationReport @Unstable public abstract void setApplicationType(String applicationType); + /** + * Get all tags corresponding to the application + * @return Application's tags + */ + @Public + @Stable + public abstract Set getApplicationTags(); + + @Private + @Unstable + public abstract void setApplicationTags(Set tags); + @Private @Stable public abstract void setAMRMToken(Token amRmToken); Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java Sat Feb 8 19:05:12 2014 @@ -25,8 +25,11 @@ import org.apache.hadoop.classification. import org.apache.hadoop.classification.InterfaceStability.Unstable; import org.apache.hadoop.yarn.api.ApplicationClientProtocol; import org.apache.hadoop.yarn.api.ApplicationMasterProtocol; +import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.util.Records; +import java.util.Set; + /** *

ApplicationSubmissionContext represents all of the * information needed by the ResourceManager to launch @@ -284,7 +287,6 @@ public abstract class ApplicationSubmiss @Stable public abstract void setApplicationType(String applicationType); - /** * Get the flag which indicates whether to keep containers across application * attempts or not. @@ -314,4 +316,26 @@ public abstract class ApplicationSubmiss @Stable public abstract void setKeepContainersAcrossApplicationAttempts( boolean keepContainers); + + /** + * Get tags for the application + * + * @return the application tags + */ + @Public + @Stable + public abstract Set getApplicationTags(); + + /** + * Set tags for the application. A maximum of + * {@link YarnConfiguration#APPLICATION_MAX_TAGS} are allowed + * per application. Each tag can be at most + * {@link YarnConfiguration#APPLICATION_MAX_TAG_LENGTH} + * characters, and can contain only ASCII characters. + * + * @param tags tags to set + */ + @Public + @Stable + public abstract void setApplicationTags(Set tags); } \ No newline at end of file Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java Sat Feb 8 19:05:12 2014 @@ -21,10 +21,13 @@ package org.apache.hadoop.yarn.conf; import com.google.common.annotations.VisibleForTesting; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.HadoopIllegalArgumentException; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; +import java.net.InetSocketAddress; import java.util.Collection; @InterfaceAudience.Private @@ -98,21 +101,7 @@ public class HAUtil { for (String id: ids) { // verify the RM service addresses configurations for every RMIds for (String prefix : YarnConfiguration.RM_SERVICES_ADDRESS_CONF_KEYS) { - String confKey = null; - try { - confKey = addSuffix(prefix, id); - if (conf.getTrimmed(confKey) == null) { - throwBadConfigurationException(getNeedToSetValueMessage(confKey)); - } - } catch (IllegalArgumentException iae) { - String errmsg = iae.getMessage(); - if (confKey == null) { - // Error at addSuffix - errmsg = getInvalidValueMessage(YarnConfiguration.RM_HA_ID, - getRMHAId(conf)); - } - throwBadConfigurationException(errmsg); - } + checkAndSetRMRPCAddress(prefix, id, conf); } setValue.append(id); setValue.append(","); @@ -122,10 +111,18 @@ public class HAUtil { } private static void verifyAndSetCurrentRMHAId(Configuration conf) { - String rmId = conf.getTrimmed(YarnConfiguration.RM_HA_ID); + String rmId = getRMHAId(conf); if (rmId == null) { - throwBadConfigurationException( - getNeedToSetValueMessage(YarnConfiguration.RM_HA_ID)); + StringBuilder msg = new StringBuilder(); + msg.append("Can not find valid RM_HA_ID. None of "); + for (String id : conf + .getTrimmedStringCollection(YarnConfiguration.RM_HA_IDS)) { + msg.append(addSuffix(YarnConfiguration.RM_ADDRESS, id) + " "); + } + msg.append(" are matching" + + " the local address OR " + YarnConfiguration.RM_HA_ID + " is not" + + " specified in HA Configuration"); + throwBadConfigurationException(msg.toString()); } else { Collection ids = getRMHAIds(conf); if (!ids.contains(rmId)) { @@ -179,7 +176,34 @@ public class HAUtil { * @return RM Id on success */ public static String getRMHAId(Configuration conf) { - return conf.get(YarnConfiguration.RM_HA_ID); + int found = 0; + String currentRMId = conf.getTrimmed(YarnConfiguration.RM_HA_ID); + if(currentRMId == null) { + for(String rmId : getRMHAIds(conf)) { + String key = addSuffix(YarnConfiguration.RM_ADDRESS, rmId); + String addr = conf.get(key); + if (addr == null) { + continue; + } + InetSocketAddress s; + try { + s = NetUtils.createSocketAddr(addr); + } catch (Exception e) { + LOG.warn("Exception in creating socket address " + addr, e); + continue; + } + if (!s.isUnresolved() && NetUtils.isLocalAddress(s.getAddress())) { + currentRMId = rmId.trim(); + found++; + } + } + } + if (found > 1) { // Only one address must match the local address + String msg = "The HA Configuration has multiple addresses that match " + + "local node's address."; + throw new HadoopIllegalArgumentException(msg); + } + return currentRMId; } @VisibleForTesting @@ -212,9 +236,13 @@ public class HAUtil { @InterfaceAudience.Private @VisibleForTesting static String getConfKeyForRMInstance(String prefix, Configuration conf) { - return YarnConfiguration.RM_SERVICES_ADDRESS_CONF_KEYS.contains(prefix) - ? addSuffix(prefix, getRMHAId(conf)) - : prefix; + if (!YarnConfiguration.RM_SERVICES_ADDRESS_CONF_KEYS.contains(prefix)) { + return prefix; + } else { + String RMId = getRMHAId(conf); + checkAndSetRMRPCAddress(prefix, RMId, conf); + return addSuffix(prefix, RMId); + } } public static String getConfValueForRMInstance(String prefix, @@ -247,4 +275,30 @@ public class HAUtil { } return key + "." + suffix; } + + private static void checkAndSetRMRPCAddress(String prefix, String RMId, + Configuration conf) { + String rpcAddressConfKey = null; + try { + rpcAddressConfKey = addSuffix(prefix, RMId); + if (conf.getTrimmed(rpcAddressConfKey) == null) { + String hostNameConfKey = addSuffix(YarnConfiguration.RM_HOSTNAME, RMId); + String confVal = conf.getTrimmed(hostNameConfKey); + if (confVal == null) { + throwBadConfigurationException(getNeedToSetValueMessage( + hostNameConfKey + " or " + addSuffix(prefix, RMId))); + } else { + conf.set(addSuffix(prefix, RMId), confVal + ":" + + YarnConfiguration.getRMDefaultPortNumber(prefix)); + } + } + } catch (IllegalArgumentException iae) { + String errmsg = iae.getMessage(); + if (rpcAddressConfKey == null) { + // Error at addSuffix + errmsg = getInvalidValueMessage(YarnConfiguration.RM_HA_ID, RMId); + } + throwBadConfigurationException(errmsg); + } + } } Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java Sat Feb 8 19:05:12 2014 @@ -23,8 +23,10 @@ import java.util.Arrays; import java.util.Collections; import java.util.List; +import org.apache.hadoop.HadoopIllegalArgumentException; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.classification.InterfaceStability.Evolving; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeysPublic; @@ -37,8 +39,26 @@ import org.apache.hadoop.yarn.api.Applic @Evolving public class YarnConfiguration extends Configuration { + @Private + public static final String CS_CONFIGURATION_FILE= "capacity-scheduler.xml"; + + @Private + public static final String HADOOP_POLICY_CONFIGURATION_FILE = + "hadoop-policy.xml"; + + @Private + public static final String YARN_SITE_XML_FILE = "yarn-site.xml"; + + @Private + public static final String CORE_SITE_CONFIGURATION_FILE = "core-site.xml"; + + @Evolving + public static final int APPLICATION_MAX_TAGS = 10; + + @Evolving + public static final int APPLICATION_MAX_TAG_LENGTH = 100; + private static final String YARN_DEFAULT_XML_FILE = "yarn-default.xml"; - private static final String YARN_SITE_XML_FILE = "yarn-site.xml"; static { Configuration.addDefaultResource(YARN_DEFAULT_XML_FILE); @@ -89,6 +109,8 @@ public class YarnConfiguration extends C public static final String RM_CLUSTER_ID = RM_PREFIX + "cluster-id"; + public static final String RM_HOSTNAME = RM_PREFIX + "hostname"; + /** The address of the applications manager interface in the RM.*/ public static final String RM_ADDRESS = RM_PREFIX + "address"; @@ -329,6 +351,16 @@ public class YarnConfiguration extends C public static final String RM_HA_IDS = RM_HA_PREFIX + "rm-ids"; public static final String RM_HA_ID = RM_HA_PREFIX + "id"; + /** Store the related configuration files in File System */ + public static final String FS_BASED_RM_CONF_STORE = RM_PREFIX + + "configuration.file-system-based-store"; + public static final String DEFAULT_FS_BASED_RM_CONF_STORE = "/yarn/conf"; + + public static final String RM_CONFIGURATION_PROVIDER_CLASS = RM_PREFIX + + "configuration.provider-class"; + public static final String DEFAULT_RM_CONFIGURATION_PROVIDER_CLASS = + "org.apache.hadoop.yarn.LocalConfigurationProvider"; + @Private public static final List RM_SERVICES_ADDRESS_CONF_KEYS = Collections.unmodifiableList(Arrays.asList( @@ -344,11 +376,11 @@ public class YarnConfiguration extends C public static final String AUTO_FAILOVER_ENABLED = AUTO_FAILOVER_PREFIX + "enabled"; - public static final boolean DEFAULT_AUTO_FAILOVER_ENABLED = false; + public static final boolean DEFAULT_AUTO_FAILOVER_ENABLED = true; public static final String AUTO_FAILOVER_EMBEDDED = AUTO_FAILOVER_PREFIX + "embedded"; - public static final boolean DEFAULT_AUTO_FAILOVER_EMBEDDED = false; + public static final boolean DEFAULT_AUTO_FAILOVER_EMBEDDED = true; public static final String AUTO_FAILOVER_ZK_BASE_PATH = AUTO_FAILOVER_PREFIX + "zk-base-path"; @@ -597,6 +629,7 @@ public class YarnConfiguration extends C */ public static final String NM_LOG_RETAIN_SECONDS = NM_PREFIX + "log.retain-seconds"; + public static final long DEFAULT_NM_LOG_RETAIN_SECONDS = 3 * 60 * 60; /** * Number of threads used in log cleanup. Only applicable if Log aggregation @@ -1000,6 +1033,19 @@ public class YarnConfiguration extends C AHS_PREFIX + "webapp.spnego-keytab-file"; //////////////////////////////// + // ATS Configs + //////////////////////////////// + + public static final String ATS_PREFIX = YARN_PREFIX + "ats."; + + /** ATS store class */ + public static final String ATS_STORE = ATS_PREFIX + "store.class"; + + /** ATS leveldb path */ + public static final String ATS_LEVELDB_PATH_PROPERTY = + ATS_PREFIX + "leveldb-apptimeline-store.path"; + + //////////////////////////////// // Other Configs //////////////////////////////// @@ -1101,4 +1147,27 @@ public class YarnConfiguration extends C } return super.updateConnectAddr(prefix, addr); } + + @Private + public static int getRMDefaultPortNumber(String addressPrefix) { + if (addressPrefix.equals(YarnConfiguration.RM_ADDRESS)) { + return YarnConfiguration.DEFAULT_RM_PORT; + } else if (addressPrefix.equals(YarnConfiguration.RM_SCHEDULER_ADDRESS)) { + return YarnConfiguration.DEFAULT_RM_SCHEDULER_PORT; + } else if (addressPrefix.equals(YarnConfiguration.RM_WEBAPP_ADDRESS)) { + return YarnConfiguration.DEFAULT_RM_WEBAPP_PORT; + } else if (addressPrefix.equals(YarnConfiguration.RM_WEBAPP_HTTPS_ADDRESS)) { + return YarnConfiguration.DEFAULT_RM_WEBAPP_HTTPS_PORT; + } else if (addressPrefix + .equals(YarnConfiguration.RM_RESOURCE_TRACKER_ADDRESS)) { + return YarnConfiguration.DEFAULT_RM_RESOURCE_TRACKER_PORT; + } else if (addressPrefix.equals(YarnConfiguration.RM_ADMIN_ADDRESS)) { + return YarnConfiguration.DEFAULT_RM_ADMIN_PORT; + } else { + throw new HadoopIllegalArgumentException( + "Invalid RM RPC address Prefix: " + addressPrefix + + ". The valid value should be one of " + + YarnConfiguration.RM_SERVICES_ADDRESS_CONF_KEYS); + } + } } Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto Sat Feb 8 19:05:12 2014 @@ -190,6 +190,7 @@ message ApplicationReportProto { optional float progress = 17; optional string applicationType = 18; optional hadoop.common.TokenProto am_rm_token = 19; + repeated string applicationTags = 20; } message ApplicationAttemptReportProto { @@ -287,6 +288,7 @@ message ApplicationSubmissionContextProt optional ResourceProto resource = 9; optional string applicationType = 10 [default = "YARN"]; optional bool keep_containers_across_application_attempts = 11 [default = false]; + repeated string applicationTags = 12; } enum ApplicationAccessTypeProto { Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto Sat Feb 8 19:05:12 2014 @@ -136,6 +136,12 @@ message MoveApplicationAcrossQueuesReque message MoveApplicationAcrossQueuesResponseProto { } +enum ApplicationsRequestScopeProto { + ALL = 0; + VIEWABLE = 1; + OWN = 2; +} + message GetApplicationsRequestProto { repeated string application_types = 1; repeated YarnApplicationStateProto application_states = 2; @@ -146,6 +152,8 @@ message GetApplicationsRequestProto { optional int64 start_end = 7; optional int64 finish_begin = 8; optional int64 finish_end = 9; + repeated string applicationTags = 10; + optional ApplicationsRequestScopeProto scope = 11 [default = ALL]; } message GetApplicationsResponseProto { Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java Sat Feb 8 19:05:12 2014 @@ -232,7 +232,6 @@ public class ApplicationMaster { private static final String shellArgsPath = "shellArgs"; private volatile boolean done; - private volatile boolean success; private ByteBuffer allTokens; @@ -254,8 +253,8 @@ public class ApplicationMaster { if (!doRun) { System.exit(0); } - result = appMaster.run(); - appMaster.finish(); + appMaster.run(); + result = appMaster.finish(); } catch (Throwable t) { LOG.fatal("Error running ApplicationMaster", t); System.exit(1); @@ -480,7 +479,7 @@ public class ApplicationMaster { * @throws IOException */ @SuppressWarnings({ "unchecked" }) - public boolean run() throws YarnException, IOException { + public void run() throws YarnException, IOException { LOG.info("Starting ApplicationMaster"); Credentials credentials = @@ -561,7 +560,6 @@ public class ApplicationMaster { amRMClient.addContainerRequest(containerAsk); } numRequestedContainers.set(numTotalContainersToRequest); - return success; } @VisibleForTesting @@ -569,7 +567,8 @@ public class ApplicationMaster { return new NMCallbackHandler(this); } - protected void finish() { + @VisibleForTesting + protected boolean finish() { // wait for completion. while (!done && (numCompletedContainers.get() != numTotalContainers)) { @@ -600,7 +599,7 @@ public class ApplicationMaster { FinalApplicationStatus appStatus; String appMessage = null; - success = true; + boolean success = true; if (numFailedContainers.get() == 0 && numCompletedContainers.get() == numTotalContainers) { appStatus = FinalApplicationStatus.SUCCEEDED; @@ -621,6 +620,8 @@ public class ApplicationMaster { } amRMClient.stop(); + + return success; } private class RMCallbackHandler implements AMRMClientAsync.CallbackHandler { Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/ContainerLaunchFailAppMaster.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/ContainerLaunchFailAppMaster.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/ContainerLaunchFailAppMaster.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/ContainerLaunchFailAppMaster.java Sat Feb 8 19:05:12 2014 @@ -18,13 +18,13 @@ package org.apache.hadoop.yarn.applications.distributedshell; +import java.nio.ByteBuffer; +import java.util.Map; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.yarn.api.records.ContainerId; -import java.nio.ByteBuffer; -import java.util.Map; - public class ContainerLaunchFailAppMaster extends ApplicationMaster { private static final Log LOG = @@ -66,8 +66,8 @@ public class ContainerLaunchFailAppMaste if (!doRun) { System.exit(0); } - result = appMaster.run(); - appMaster.finish(); + appMaster.run(); + result = appMaster.finish(); } catch (Throwable t) { LOG.fatal("Error running ApplicationMaster", t); System.exit(1); Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java Sat Feb 8 19:05:12 2014 @@ -29,8 +29,8 @@ public class TestDSFailedAppMaster exten private static final Log LOG = LogFactory.getLog(TestDSFailedAppMaster.class); @Override - public boolean run() throws YarnException, IOException { - boolean res = super.run(); + public void run() throws YarnException, IOException { + super.run(); // for the 2nd attempt. if (appAttemptID.getAttemptId() == 2) { @@ -39,11 +39,12 @@ public class TestDSFailedAppMaster exten // numRequestedContainers should be set to 0. if (numAllocatedContainers.get() != 1 || numRequestedContainers.get() != 0) { - LOG.info("Application Master failed. exiting"); + LOG.info("NumAllocatedContainers is " + numAllocatedContainers.get() + + " and NumRequestedContainers is " + numAllocatedContainers.get() + + ".Application Master failed. exiting"); System.exit(200); } } - return res; } public static void main(String[] args) { @@ -54,7 +55,7 @@ public class TestDSFailedAppMaster exten if (!doRun) { System.exit(0); } - result = appMaster.run(); + appMaster.run(); if (appMaster.appAttemptID.getAttemptId() == 1) { try { // sleep some time, wait for the AM to launch a container. @@ -63,7 +64,7 @@ public class TestDSFailedAppMaster exten // fail the first am. System.exit(100); } - appMaster.finish(); + result = appMaster.finish(); } catch (Throwable t) { System.exit(1); } Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java Sat Feb 8 19:05:12 2014 @@ -436,4 +436,19 @@ public abstract class YarnClient extends public abstract List getContainers( ApplicationAttemptId applicationAttemptId) throws YarnException, IOException; + + /** + *

+ * Attempts to move the given application to the given queue. + *

+ * + * @param appId + * Application to move. + * @param queue + * Queue to place it in to. + * @throws YarnException + * @throws IOException + */ + public abstract void moveApplicationAcrossQueues(ApplicationId appId, + String queue) throws YarnException, IOException; } Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java Sat Feb 8 19:05:12 2014 @@ -48,6 +48,7 @@ import org.apache.hadoop.yarn.api.protoc import org.apache.hadoop.yarn.api.protocolrecords.GetQueueUserAclsInfoRequest; import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationRequest; import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationResponse; +import org.apache.hadoop.yarn.api.protocolrecords.MoveApplicationAcrossQueuesRequest; import org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationRequest; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; @@ -478,4 +479,12 @@ public class YarnClientImpl extends Yarn } throw new YarnException("History service is not enabled."); } + + @Override + public void moveApplicationAcrossQueues(ApplicationId appId, + String queue) throws YarnException, IOException { + MoveApplicationAcrossQueuesRequest request = + MoveApplicationAcrossQueuesRequest.newInstance(appId, queue); + rmClient.moveApplicationAcrossQueues(request); + } } Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java Sat Feb 8 19:05:12 2014 @@ -61,6 +61,7 @@ public class ApplicationCLI extends Yarn private static final String APP_TYPE_CMD = "appTypes"; private static final String APP_STATE_CMD = "appStates"; private static final String ALLSTATES_OPTION = "ALL"; + private static final String QUEUE_CMD = "queue"; public static final String APPLICATION = "application"; public static final String APPLICATION_ATTEMPT = "applicationattempt"; public static final String CONTAINER = "container"; @@ -96,6 +97,10 @@ public class ApplicationCLI extends Yarn + "and -appStates to filter applications based on application state"); } opts.addOption(KILL_CMD, true, "Kills the application."); + opts.addOption(MOVE_TO_QUEUE_CMD, true, "Moves the application to a " + + "different queue."); + opts.addOption(QUEUE_CMD, true, "Works with the movetoqueue command to" + + " specify which queue to move an application to."); opts.addOption(HELP_CMD, false, "Displays help for all commands."); Option appTypeOpt = new Option(APP_TYPE_CMD, true, "Works with -list to " + "filter applications based on " @@ -112,6 +117,8 @@ public class ApplicationCLI extends Yarn appStateOpt.setArgName("States"); opts.addOption(appStateOpt); opts.getOption(KILL_CMD).setArgName("Application ID"); + opts.getOption(MOVE_TO_QUEUE_CMD).setArgName("Application ID"); + opts.getOption(QUEUE_CMD).setArgName("Queue Name"); opts.getOption(STATUS_CMD).setArgName("Application ID"); int exitCode = -1; @@ -197,11 +204,18 @@ public class ApplicationCLI extends Yarn listApplications(appTypes, appStates); } } else if (cliParser.hasOption(KILL_CMD)) { - if (args.length != 2) { + if (args.length != 3) { printUsage(opts); return exitCode; } killApplication(cliParser.getOptionValue(KILL_CMD)); + } else if (cliParser.hasOption(MOVE_TO_QUEUE_CMD)) { + if (!cliParser.hasOption(QUEUE_CMD)) { + printUsage(opts); + return exitCode; + } + moveApplicationAcrossQueues(cliParser.getOptionValue(MOVE_TO_QUEUE_CMD), + cliParser.getOptionValue(QUEUE_CMD)); } else if (cliParser.hasOption(HELP_CMD)) { printUsage(opts); return 0; @@ -366,6 +380,28 @@ public class ApplicationCLI extends Yarn client.killApplication(appId); } } + + /** + * Kills the application with the application id as appId + * + * @param applicationId + * @throws YarnException + * @throws IOException + */ + private void moveApplicationAcrossQueues(String applicationId, String queue) + throws YarnException, IOException { + ApplicationId appId = ConverterUtils.toApplicationId(applicationId); + ApplicationReport appReport = client.getApplicationReport(appId); + if (appReport.getYarnApplicationState() == YarnApplicationState.FINISHED + || appReport.getYarnApplicationState() == YarnApplicationState.KILLED + || appReport.getYarnApplicationState() == YarnApplicationState.FAILED) { + sysout.println("Application " + applicationId + " has already finished "); + } else { + sysout.println("Moving application " + applicationId + " to queue " + queue); + client.moveApplicationAcrossQueues(appId, queue); + sysout.println("Successfully completed move."); + } + } /** * Prints the application report for an application id. Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java Sat Feb 8 19:05:12 2014 @@ -33,6 +33,7 @@ public abstract class YarnCLI extends Co public static final String STATUS_CMD = "status"; public static final String LIST_CMD = "list"; public static final String KILL_CMD = "kill"; + public static final String MOVE_TO_QUEUE_CMD = "movetoqueue"; public static final String HELP_CMD = "help"; protected PrintStream sysout; protected PrintStream syserr; Modified: hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java?rev=1566100&r1=1566099&r2=1566100&view=diff ============================================================================== --- hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java (original) +++ hadoop/common/branches/HDFS-4685/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java Sat Feb 8 19:05:12 2014 @@ -172,8 +172,6 @@ public class TestRMFailover extends Clie @Test public void testAutomaticFailover() throws YarnException, InterruptedException, IOException { - conf.setBoolean(YarnConfiguration.AUTO_FAILOVER_ENABLED, true); - conf.setBoolean(YarnConfiguration.AUTO_FAILOVER_EMBEDDED, true); conf.set(YarnConfiguration.RM_CLUSTER_ID, "yarn-test-cluster"); conf.set(YarnConfiguration.RM_ZK_ADDRESS, hostPort); conf.setInt(YarnConfiguration.RM_ZK_TIMEOUT_MS, 2000); @@ -193,6 +191,7 @@ public class TestRMFailover extends Clie @Test public void testWebAppProxyInStandAloneMode() throws YarnException, InterruptedException, IOException { + conf.setBoolean(YarnConfiguration.AUTO_FAILOVER_ENABLED, false); WebAppProxyServer webAppProxyServer = new WebAppProxyServer(); try { conf.set(YarnConfiguration.PROXY_ADDRESS, "0.0.0.0:9099"); @@ -227,6 +226,7 @@ public class TestRMFailover extends Clie @Test public void testEmbeddedWebAppProxy() throws YarnException, InterruptedException, IOException { + conf.setBoolean(YarnConfiguration.AUTO_FAILOVER_ENABLED, false); cluster.init(conf); cluster.start(); getAdminService(0).transitionToActive(req);