hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Luo <j...@merkleinc.com>
Subject RE: how to use Yarn API to find task/attempt status
Date Thu, 10 Mar 2016 17:21:41 GMT
Thanks David/Jeff.

To avoid further confusions, let me make sure I am clear on what I am trying to do: I would
like to know how many hours in a day my cluster is running at its full capacity, and when
that happens, how long is my waiting queue. I founded similar information on Ambari as below,
but I’d like to dive deeper, hence asking.

From what I see, container per job information, especially pending containers, is only available
from an application’s trackingUrl, but that just applies to M/R jobs. I am not able to get
the same information from a Tez applications’ trackingUrl (Tez’s url doesn’t do anything
for hdp2.2).  So how does Ambari find the information out?


[cid:image001.png@01D17ABE.06C936C0]

From: David Morel [mailto:dmorel@amakuru.net]
Sent: Thursday, March 10, 2016 1:03 AM
To: Jeff Zhang
Cc: user@hadoop.apache.org; Frank Luo
Subject: Re: how to use Yarn API to find task/attempt status


The REST API should help. A working implementation (in perl, not java, sorry) is visible here
: http://search.cpan.org/dist/Net-Hadoop-YARN/
Read the comments, they matter :-)
Le 10 mars 2016 7:28 AM, "Jeff Zhang" <zjffdu@gmail.com<mailto:zjffdu@gmail.com>>
a écrit :
If it is for M/R, then maybe this is what you want
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/mapreduce/JobStatus.html



On Thu, Mar 10, 2016 at 1:58 PM, Frank Luo <jluo@merkleinc.com<mailto:jluo@merkleinc.com>>
wrote:
Let’s say there are 10 standard M/R jobs running. How to find how many tasks are done/running/pending?

From: Jeff Zhang [mailto:zjffdu@gmail.com<mailto:zjffdu@gmail.com>]
Sent: Wednesday, March 09, 2016 9:33 PM
To: Frank Luo
Cc: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: how to use Yarn API to find task/attempt status

I don't think it is related with yarn. Yarn don't know about task/task attempt, it only knows
containers. So it should be your application to provide such function.

On Thu, Mar 10, 2016 at 11:29 AM, Frank Luo <jluo@merkleinc.com<mailto:jluo@merkleinc.com>>
wrote:
Anyone had a similar issue and knows the answer?

From: Frank Luo
Sent: Wednesday, March 09, 2016 1:59 PM
To: 'user@hadoop.apache.org<mailto:user@hadoop.apache.org>'
Subject: how to use Yarn API to find task/attempt status

I have a need to programmatically find out how many tasks are pending in Yarn. Is there a
way to do it through a Java API?

I looked at YarnClient, but not able to find what I need.

Thx in advance.

Frank Luo

This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.



--
Best Regards

Jeff Zhang

This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.



--
Best Regards

Jeff Zhang

This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.
Mime
View raw message