Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BE4562009FB for ; Fri, 6 May 2016 14:45:14 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BCDCC160A0E; Fri, 6 May 2016 12:45:14 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0EEDE1609F6 for ; Fri, 6 May 2016 14:45:13 +0200 (CEST) Received: (qmail 33820 invoked by uid 500); 6 May 2016 12:45:13 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 33786 invoked by uid 99); 6 May 2016 12:45:13 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2016 12:45:13 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 06D892C1F64 for ; Fri, 6 May 2016 12:45:13 +0000 (UTC) Date: Fri, 6 May 2016 12:45:13 +0000 (UTC) From: "Jason Lowe (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-5039) Applications ACCEPTED but not starting MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 06 May 2016 12:45:14 -0000 [ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273984#comment-15273984 ] Jason Lowe commented on YARN-5039: ---------------------------------- bq. scheduler will not assign containers to decommissioning nodes, that could be the reason why your applications stay at ACCEPTED state. When I saw those log messages I immediately thought that was the case, but I couldn't see any of the three completely empty nodes in the list of nodes that supposedly were decommissioning. In addition the debug logs clearly show the nodes are heartbeating in, the nodes page shows the RM thinks the nodes have 256GB available, and as Miles mentioned the nodes are immediately used when the second app's AM finally starts. Therefore I don't think this is related to node decommissioning unless the Amazon node decommissioning logic is very bizarre and somehow tied to when applications start. > Applications ACCEPTED but not starting > -------------------------------------- > > Key: YARN-5039 > URL: https://issues.apache.org/jira/browse/YARN-5039 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.7.2 > Reporter: Miles Crawford > Attachments: Screen Shot 2016-05-04 at 1.57.19 PM.png, Screen Shot 2016-05-04 at 2.41.22 PM.png, queue-config.log, resource-manager-application-starts.log.gz, yarn-yarn-resourcemanager-ip-10-12-47-144.log.gz > > > Often when we submit applications to an incompletely utilized cluster, they sit, unable to start for no apparent reason. > There are multiple nodes in the cluster with available resources, but the resourcemanger logs show that scheduling is being skipped. The scheduling is skipped because the application itself has reserved the node? I'm not sure how to interpret this log output: > {code} > 2016-05-04 20:19:21,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Trying to fulfill reservation for application application_1462291866507_0025 on node: ip-10-12-43-54.us-west-2.compute.internal:8041 > 2016-05-04 20:19:21,316 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Reserved container application=application_1462291866507_0025 resource= queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=, usedCapacity=0.7126589, absoluteUsedCapacity=0.7126589, numApps=2, numContainers=33 usedCapacity=0.7126589 absoluteUsedCapacity=0.7126589 used= cluster= > 2016-05-04 20:19:21,316 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Skipping scheduling since node ip-10-12-43-54.us-west-2.compute.internal:8041 is reserved by application appattempt_1462291866507_0025_000001 > 2016-05-04 20:19:22,232 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Trying to fulfill reservation for application application_1462291866507_0025 on node: ip-10-12-43-53.us-west-2.compute.internal:8041 > 2016-05-04 20:19:22,232 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Reserved container application=application_1462291866507_0025 resource= queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=, usedCapacity=0.7126589, absoluteUsedCapacity=0.7126589, numApps=2, numContainers=33 usedCapacity=0.7126589 absoluteUsedCapacity=0.7126589 used= cluster= > 2016-05-04 20:19:22,232 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Skipping scheduling since node ip-10-12-43-53.us-west-2.compute.internal:8041 is reserved by application appattempt_1462291866507_0025_000001 > 2016-05-04 20:19:22,316 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Trying to fulfill reservation for application application_1462291866507_0025 on node: ip-10-12-43-54.us-west-2.compute.internal:8041 > 2016-05-04 20:19:22,316 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Reserved container application=application_1462291866507_0025 resource= queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=, usedCapacity=0.7126589, absoluteUsedCapacity=0.7126589, numApps=2, numContainers=33 usedCapacity=0.7126589 absoluteUsedCapacity=0.7126589 used= cluster= > 2016-05-04 20:19:22,316 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Skipping scheduling since node ip-10-12-43-54.us-west-2.compute.internal:8041 is reserved by application appattempt_1462291866507_0025_000001 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org