Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8F7F5200B78 for ; Fri, 19 Aug 2016 06:14:09 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8DFE0160AB7; Fri, 19 Aug 2016 04:14:09 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6273B160AAE for ; Fri, 19 Aug 2016 06:14:08 +0200 (CEST) Received: (qmail 21433 invoked by uid 500); 19 Aug 2016 04:14:06 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 21418 invoked by uid 99); 19 Aug 2016 04:14:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Aug 2016 04:14:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 81112C0127 for ; Fri, 19 Aug 2016 04:14:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.2 X-Spam-Level: *** X-Spam-Status: No, score=3.2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id RaAd6MwgpVdr for ; Fri, 19 Aug 2016 04:14:03 +0000 (UTC) Received: from mail-io0-f181.google.com (mail-io0-f181.google.com [209.85.223.181]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 41C6F5F4E5 for ; Fri, 19 Aug 2016 04:14:02 +0000 (UTC) Received: by mail-io0-f181.google.com with SMTP id b62so37978933iod.3 for ; Thu, 18 Aug 2016 21:14:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=jxQX3powStsl+iozGlED40wH7VFUnF2iaZcP78ctfOg=; b=in4i4xLp2Ntt+ZiuAk9IkaYlb9OHmE/JrPuEbcPjTteUKmvmutGpQE24/VPrTGzDDL PYTBpblxrCccGhy8Y3tijlp7TGPLpVGsmqz4mcbQpiiO3PNrGBRIPQRHA5pISOTic7jU RK66gxUwDouGUuLJJxjdW5nmnq0CDJIMO5lgJkIaBMRvpBYtdu96L9HaasRo/T8uIm4o oHJgIKN3mupp2sMhXSC+mAcNXpvAylgsiApoU0EVv+TVzbKfJRzZV6kDjqeTxPNaywG8 P64RmMzJZNqtcdwCSRyhqVl0WFkXuVGpKzcn2/5Ibo1FWlg2gnalFwIZLWmT83RUutEQ IDkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=jxQX3powStsl+iozGlED40wH7VFUnF2iaZcP78ctfOg=; b=aXothNrHHUrdjSl3wGG/wwOP7nxqkHR+xIxSpbx3g7vjffz1y3LTCN/w4PqE79RDff eeDBMuezCyGmlRi7aQVH1yKIrisS2N6FV789VvQqLQ3nxeYd+napByAKVcUwBMICOsnL ElquCH6iLXLGCr9NqdhEsmj0QqWv0UTLX7unVBjh3T8ZZbPA24arG+wHf2hl3GwDW40V SKZbmSWjsZfRJ27Umezr++jOA2X3yA3ugO4skVkOx1+dNxcyeYiq9/TbRopm9BMfysjJ qGMgNg2G2ls8MRUP17dxD7yc4gT8RwShVuDapxoiM/wvXmXeQsE1DdqiCbzqgeneE4g5 aRog== X-Gm-Message-State: AEkoout5Fn+aLrMqGRjXdVL5vSMVKmF6FRok9RSu8Q9MAiRyLiOU5RaFPeiym9l982JhrQqRw7EruxRqSbBmTQ== X-Received: by 10.107.40.133 with SMTP id o127mr7040665ioo.183.1471580041220; Thu, 18 Aug 2016 21:14:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.66.210 with HTTP; Thu, 18 Aug 2016 21:13:20 -0700 (PDT) In-Reply-To: <57B68705.7010508@gmail.com> References: <57B67B16.5040202@gmail.com> <57B68705.7010508@gmail.com> From: rammohan ganapavarapu Date: Thu, 18 Aug 2016 21:13:20 -0700 Message-ID: Subject: Re: ACCEPTED: waiting for AM container to be allocated, launched and register with RM To: tkg_cangkul Cc: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11352e2a9d9b6d053a64ed9f archived-at: Fri, 19 Aug 2016 04:14:09 -0000 --001a11352e2a9d9b6d053a64ed9f Content-Type: text/plain; charset=UTF-8 Do you know what properties to tune? Thanks, Ram On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul wrote: > i think that's because you don't have enough resource. u can tune your > cluster config to maximize your resource. > > > On 19/08/16 11:03, rammohan ganapavarapu wrote: > > I dont see any thing odd except this not sure if i have to worry about it > or not. > > 2016-08-19 03:29:26,621 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at /0.0.0.0:8030 > 2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client: Retrying > connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry > policy is RetryUpToMaximumCo > untWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) > 2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client: Retrying > connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, > sleepTime=1000 MILLISECONDS) > > > its keep printing this log ..in app container logs. > > On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul > wrote: > >> maybe u can check the logs from port 8088 on your browser. that was RM >> UI. just choose your job id and then check the logs. >> >> On 19/08/16 10:14, rammohan ganapavarapu wrote: >> >> Sunil, >> >> Thanks you for your input, below are my server metrics for RM. Also >> attached RM UI for capacity scheduler resources. How else i can find? >> >> { >> "name": "Hadoop:service=ResourceManager,name=QueueMetrics,q0=root", >> "modelerType": "QueueMetrics,q0=root", >> "tag.Queue": "root", >> "tag.Context": "yarn", >> "tag.Hostname": "hadoop001", >> "running_0": 0, >> "running_60": 0, >> "running_300": 0, >> "running_1440": 0, >> "AppsSubmitted": 1, >> "AppsRunning": 0, >> "AppsPending": 0, >> "AppsCompleted": 0, >> "AppsKilled": 0, >> "AppsFailed": 1, >> "AllocatedMB": 0, >> "AllocatedVCores": 0, >> "AllocatedContainers": 0, >> "AggregateContainersAllocated": 2, >> "AggregateContainersReleased": 2, >> "AvailableMB": 64512, >> "AvailableVCores": 24, >> "PendingMB": 0, >> "PendingVCores": 0, >> "PendingContainers": 0, >> "ReservedMB": 0, >> "ReservedVCores": 0, >> "ReservedContainers": 0, >> "ActiveUsers": 0, >> "ActiveApplications": 0 >> }, >> >> On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind >> wrote: >> >>> Hi >>> >>> It could be because of many of reasons. Also I am not sure about which >>> scheduler your are using, pls share more details such as RM log etc. >>> >>> I could point out few reasons >>> - Such as "Not enough resource is cluster" can cause this >>> - If using Capacity Scheduler, if queue capacity is maxed out, such >>> case can happen. >>> - Similarly if max-am-resource-percent is crossed per queue level, then >>> also AM container may not be launched. >>> >>> you could check RM log to get more information if AM container is >>> laucnhed. >>> >>> Thanks >>> Sunil >>> >>> On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu < >>> rammohanganap@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> When i submit a MR job, i am getting this from AM UI but it never get >>>> finished, what am i missing ? >>>> >>>> Thanks, >>>> Ram >>>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org >> For additional commands, e-mail: user-help@hadoop.apache.org >> >> >> > > --001a11352e2a9d9b6d053a64ed9f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Do you know what properties to tune?

Thanks,
Ram

On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul <yuza.ra= sfar@gmail.com> wrote:
=20 =20 =20
i think that's because you don't have enough resource.=C2=A0 u = can tune your cluster config to maximize your resource.

On 19/08/16 11:03, rammohan ganapavarapu wrote:
I dont see any thing odd except this not sure if i have to worry about it or not.

2016-08-19 03:29:26,621 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030= . Already tried 0 time(s); retry policy is RetryUpToMaximumCo
untWithFixedSleep(maxRetries=3D10, sleepTime=3D1000 MILLISEC= ONDS)
2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030= . Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sleepTime=3D1000 MILLISECONDS)


its keep printing this log ..in app container logs.

On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul <yuza.rasfar@gmail.com> wrote:
maybe u can check th= e logs from port 8088 on your browser. that was RM UI. just choose your job id and then check the logs.
=C2=A0
On 19/08/16 10:14, rammohan ganapavarapu wrote:
Sunil,

Thanks you for your input, below are my server metrics for RM. Also attached RM UI for capacity scheduler resources. How else i can find?

{
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "name": &quo= t;Hadoop:service=3DResourceManager,name=3DQueueMetrics,q0=3Droot"= ,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "modelerType"= ;: "QueueMetrics,q0=3Droot",
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "tag.Queue":= "root",
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "tag.Context"= ;: "yarn",
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "tag.Hostname&quo= t;: "hadoop001",
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "running_0":= 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "running_60"= : 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "running_300"= ;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "running_1440&quo= t;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AppsSubmitted&qu= ot;: 1,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AppsRunning"= ;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AppsPending"= ;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AppsCompleted&qu= ot;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AppsKilled"= : 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AppsFailed"= : 1,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AllocatedMB"= ;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AllocatedVCores&= quot;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AllocatedContain= ers": 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AggregateContain= ersAllocated": 2,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AggregateContain= ersReleased": 2,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AvailableMB"= ;: 64512,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "AvailableVCores&= quot;: 24,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "PendingMB":= 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "PendingVCores&qu= ot;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "PendingContainer= s": 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "ReservedMB"= : 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "ReservedVCores&q= uot;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "ReservedContaine= rs": 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "ActiveUsers"= ;: 0,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "ActiveApplicatio= ns": 0
=C2=A0=C2=A0=C2=A0 },

On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind <sunil.govind@gmail.com= > wrote:
Hi

It could be because of many of reasons. Also I am not sure about which scheduler your are using, pls share more details such as RM log etc.

I could point out few reasons
=C2=A0-=C2=A0Such as "Not enough resource is cluster&quo= t; can cause this
=C2=A0- If using Capacity Scheduler, if queue capacity is maxed out, such case can happen.
=C2=A0- Similarly if max-am-resource-percent is crossed per queue level, then also AM container may not be launched.

you could check RM log to get more information if AM container is laucnhed.

Thanks
Sunil<= /span>

On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu <rammohanganap@gmail.c= om> wrote:
Hi,

When i submit a MR job, i am getting this from AM UI but it never get finished, what am i missing ?

Thanks,
Ram



-------------------------------------------------=
--------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org




--001a11352e2a9d9b6d053a64ed9f--