Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
From: "David Morel" <david.morel@amakuru.net>
To: "Frank Luo" <jluo@merkleinc.com>
Cc: "David Morel" <dmorel@amakuru.net>, "Jeff Zhang" <zjffdu@gmail.com>,
 "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Re: how to use Yarn API to find task/attempt status
Date: Thu, 10 Mar 2016 23:43:40 +0100
Message-ID: <07509848-66BA-40DB-84CE-D11085FCAE67@amakuru.net>
In-Reply-To: 
 <1EFFC0EDF85B044A99A02C925FDA3755200DECA4@HQITEXCH07.pclc0.merkle.local>
References: 
 <1EFFC0EDF85B044A99A02C925FDA3755200DE06B@HQITEXCH07.pclc0.merkle.local>
 <CAADy7x7mud44swpfOGjnGV6ut49D62saoSNpYnWZUs4Ycm7HuA@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA3755200DE139@HQITEXCH07.pclc0.merkle.local>
 <CAADy7x5m2wjFe0k0_-YYFge6dhEeiYtLj_v1WrY_P6tp_rSMZQ@mail.gmail.com>
 <CAA5+UsZyEC3wk4evsjoTTk2U6pEe_5Sq7_=5CBLqh8TyFyhtZg@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA3755200DECA4@HQITEXCH07.pclc0.merkle.local>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 10 Mar 2016, at 18:21, Frank Luo wrote:

> Thanks David/Jeff.
>
> To avoid further confusions, let me make sure I am clear on what I am =

> trying to do: I would like to know how many hours in a day my cluster =

> is running at its full capacity, and when that happens, how long is my =

> waiting queue. I founded similar information on Ambari as below, but =

> I=E2=80=99d like to dive deeper, hence asking.
>
> From what I see, container per job information, especially pending =

> containers, is only available from an application=E2=80=99s trackingUrl=
, but =

> that just applies to M/R jobs. I am not able to get the same =

> information from a Tez applications=E2=80=99 trackingUrl (Tez=E2=80=99s=
 url =

> doesn=E2=80=99t do anything for hdp2.2).  So how does Ambari find the =

> information out?

Using the REST API you'd query the resource manager's "apps" method, =

then the appmasters through the RM proxy with the "jobs" method =

(sequentially, using the app ids found at step 1 in turn). Works for MR, =

there used to be an issue with spark jobs, haven't looked at that. This =

is only for running jobs; you'd probably want to query the history =

server too which may return more complete info with less indirection. =

Also, have a look at the "scheduler" method on the RM, which you may =

find useful.

The docs are here:
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/Resour=
ceManagerRest.html
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/NodeMa=
nagerRest.html

For MR stuff:
https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapr=
educe-client-core/MapredAppMasterRest.html
https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapr=
educe-client-hs/HistoryServerRest.html

But most useful is probably the timeline server, which I didn't have a =

chance to use and possibly provides what you need:
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/Timeli=
neServer.html#Timeline_Server_REST_API_v1

All this from memory since I haven't touched a cluster lately, and =

hoping it's not completely missing the point ;-)

David

>
>
> [cid:image001.png@01D17ABE.06C936C0]
>
> From: David Morel [mailto:dmorel@amakuru.net]
> Sent: Thursday, March 10, 2016 1:03 AM
> To: Jeff Zhang
> Cc: user@hadoop.apache.org; Frank Luo
> Subject: Re: how to use Yarn API to find task/attempt status
>
>
> The REST API should help. A working implementation (in perl, not java, =

> sorry) is visible here : http://search.cpan.org/dist/Net-Hadoop-YARN/
> Read the comments, they matter :-)
> Le 10 mars 2016 7:28 AM, "Jeff Zhang" =

> <zjffdu@gmail.com<mailto:zjffdu@gmail.com>> a =C3=A9crit :
> If it is for M/R, then maybe this is what you want
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/mapreduce/J=
obStatus.html
>
>
>
> On Thu, Mar 10, 2016 at 1:58 PM, Frank Luo =

> <jluo@merkleinc.com<mailto:jluo@merkleinc.com>> wrote:
> Let=E2=80=99s say there are 10 standard M/R jobs running. How to find h=
ow =

> many tasks are done/running/pending?
>
> From: Jeff Zhang [mailto:zjffdu@gmail.com<mailto:zjffdu@gmail.com>]
> Sent: Wednesday, March 09, 2016 9:33 PM
> To: Frank Luo
> Cc: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
> Subject: Re: how to use Yarn API to find task/attempt status
>
> I don't think it is related with yarn. Yarn don't know about task/task =

> attempt, it only knows containers. So it should be your application to =

> provide such function.
>
> On Thu, Mar 10, 2016 at 11:29 AM, Frank Luo =

> <jluo@merkleinc.com<mailto:jluo@merkleinc.com>> wrote:
> Anyone had a similar issue and knows the answer?
>
> From: Frank Luo
> Sent: Wednesday, March 09, 2016 1:59 PM
> To: 'user@hadoop.apache.org<mailto:user@hadoop.apache.org>'
> Subject: how to use Yarn API to find task/attempt status
>
> I have a need to programmatically find out how many tasks are pending =

> in Yarn. Is there a way to do it through a Java API?
>
> I looked at YarnClient, but not able to find what I need.
>
> Thx in advance.
>
> Frank Luo
>
> This email and any attachments transmitted with it are intended for =

> use by the intended recipient(s) only. If you have received this email =

> in error, please notify the sender immediately and then delete it. If =

> you are not the intended recipient, you must not keep, use, disclose, =

> copy or distribute this email without the author=E2=80=99s prior permis=
sion. =

> We take precautions to minimize the risk of transmitting software =

> viruses, but we advise you to perform your own virus checks on any =

> attachment to this message. We cannot accept liability for any loss or =

> damage caused by software viruses. The information contained in this =

> communication may be confidential and may be subject to the =

> attorney-client privilege.
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
> This email and any attachments transmitted with it are intended for =

> use by the intended recipient(s) only. If you have received this email =

> in error, please notify the sender immediately and then delete it. If =

> you are not the intended recipient, you must not keep, use, disclose, =

> copy or distribute this email without the author=E2=80=99s prior permis=
sion. =

> We take precautions to minimize the risk of transmitting software =

> viruses, but we advise you to perform your own virus checks on any =

> attachment to this message. We cannot accept liability for any loss or =

> damage caused by software viruses. The information contained in this =

> communication may be confidential and may be subject to the =

> attorney-client privilege.
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
> This email and any attachments transmitted with it are intended for =

> use by the intended recipient(s) only. If you have received this email =

> in error, please notify the sender immediately and then delete it. If =

> you are not the intended recipient, you must not keep, use, disclose, =

> copy or distribute this email without the author=E2=80=99s prior permis=
sion. =

> We take precautions to minimize the risk of transmitting software =

> viruses, but we advise you to perform your own virus checks on any =

> attachment to this message. We cannot accept liability for any loss or =

> damage caused by software viruses. The information contained in this =

> communication may be confidential and may be subject to the =

> attorney-client privilege.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org