hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: Task status query
Date Fri, 20 Sep 2013 12:02:13 GMT
Thanks Harsh.  Is this protocol something that is available to all AMs/tasks?  Or is it up
to each AM/task pair to develop their own protocol?

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com] 
Sent: Thursday, September 19, 2013 9:20 PM
To: <user@hadoop.apache.org>
Subject: Re: Task status query

Hi John,

YARN tasks can be more than simple executables. In case of MR, for example, tasks talk to
the AM and report their individual progress and counters back to it, via a specific protocol
(over the network), giving the AM more data to compute an near-accurate global progress.

On Fri, Sep 20, 2013 at 12:18 AM, John Lilley <john.lilley@redpoint.net> wrote:
> How does a YARN application master typically query ongoing status 
> (like percentage completion) of its tasks?
> I would like to be able to ultimately relay information to the user like:
> 100 tasks are scheduled
> 10 tasks are complete
> 4 tasks are running and they are (4%, 10%, 50%, 70%) complete
> But, given that YARN tasks are simply executables, how can the AM even 
> get at this information?  Can the AM get access to stdout/stderr?
> Thanks
> John

Harsh J

View raw message