Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A2F9710DCC for ; Mon, 23 Sep 2013 23:41:56 +0000 (UTC) Received: (qmail 89551 invoked by uid 500); 23 Sep 2013 23:41:46 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 89443 invoked by uid 500); 23 Sep 2013 23:41:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 89436 invoked by uid 99); 23 Sep 2013 23:41:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Sep 2013 23:41:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of acm@hortonworks.com designates 209.85.160.44 as permitted sender) Received: from [209.85.160.44] (HELO mail-pb0-f44.google.com) (209.85.160.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Sep 2013 23:41:40 +0000 Received: by mail-pb0-f44.google.com with SMTP id xa7so3806360pbc.17 for ; Mon, 23 Sep 2013 16:41:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :references:to:in-reply-to:content-type; bh=yA52MqxB2cc13xW7kwX+HMf+/lRv+8cpk6qwS4vIxQY=; b=KRCPH3VTM41c3wEMejP6q85bZRbXTY5C+JYldRWCvBZ1e8j4ecyinwZ7frbU++t/WJ u4qRNpcuDM2R6NsLKu3JJaksAestQyuoiuRDrXIUgON7aDq9fWM01M2cBoNTxIjCqid2 tvnA/CzRVySEB6Ho1Eml01RKEvbgWrNK3KFCgSVNj2vSE7LFs4Ae1+Dbgs+czqvRH8vg 1H2138dVzQLmuS4TrkgqXVGFeM0CNF8c8li9bzP8vVU8ExWQF9R+JaVM3gYPcDjxi/hv 0MdQOymdjuy0jxZkHkWnAawivPZOqsZm5UuJSKzkgVW4Xf+27KStS+kdODoWAKihbW/R 1eew== X-Gm-Message-State: ALoCoQnN5CqTdHE2+3jq5iQ+ZKfpwYbE2tmMHVD5hhWNfVkFPETGDAX9o/TQxpKuBez61LX0s7kcL/N4QtC+M4faJ07n6BsEnT5NUG+sXWOgHU7UtLmeyVw= X-Received: by 10.68.190.197 with SMTP id gs5mr25481570pbc.90.1379979679810; Mon, 23 Sep 2013 16:41:19 -0700 (PDT) Received: from [10.11.2.123] ([192.175.27.2]) by mx.google.com with ESMTPSA id or6sm2993239pbb.36.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 23 Sep 2013 16:41:18 -0700 (PDT) From: Arun C Murthy Message-Id: <5DEC610B-B382-4A36-B9A5-A06F0E28B061@hortonworks.com> Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Task status query Date: Mon, 23 Sep 2013 16:41:16 -0700 References: <869970D71E26D7498BDAC4E1CA92226B86D363CB@MBX021-E3-NJ-2.exch021.domain.local> <869970D71E26D7498BDAC4E1CA92226B86D37282@MBX021-E3-NJ-2.exch021.domain.local> <869970D71E26D7498BDAC4E1CA92226B86D385A5@MBX021-E3-NJ-2.exch021.domain.local> To: user@hadoop.apache.org In-Reply-To: <869970D71E26D7498BDAC4E1CA92226B86D385A5@MBX021-E3-NJ-2.exch021.domain.local> X-Mailer: Apple Mail (2.1508) Content-Type: multipart/alternative; boundary="Apple-Mail=_8724AAF5-E9B4-4E8A-96AB-DEF27CD1ED5E" X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_8724AAF5-E9B4-4E8A-96AB-DEF27CD1ED5E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1 Yep, typically, the AM should pass it's host:port to the task as part of ei= ther the cmd-line for the task or in it's env. That is what is done by MR A= M. hth, Arun On Sep 21, 2013, at 6:52 AM, John Lilley wrote: > Thanks Harsh! The data-transport format is pretty easy, but how is the R= PC typically set up? Does the AM open a listen port to accept the RPC from= the tasks, and then pass the port/URI to the tasks when they are spawned a= s command-line or environment? > john >=20 > -----Original Message----- > From: Harsh J [mailto:harsh@cloudera.com]=20 > Sent: Friday, September 20, 2013 7:47 AM > To: > Subject: Re: Task status query >=20 > Right now its MR specific (TaskUmbilicalProtocol) - YARN doesn't have any= reusable items here yet, but there are easy to use RPC libs such as Avro a= nd Thrift out there that make it easy to do such things once you define wha= t you want in a schema/spec form. >=20 > On Fri, Sep 20, 2013 at 5:32 PM, John Lilley w= rote: >> Thanks Harsh. Is this protocol something that is available to all AMs/t= asks? Or is it up to each AM/task pair to develop their own protocol? >> john >>=20 >> -----Original Message----- >> From: Harsh J [mailto:harsh@cloudera.com] >> Sent: Thursday, September 19, 2013 9:20 PM >> To: >> Subject: Re: Task status query >>=20 >> Hi John, >>=20 >> YARN tasks can be more than simple executables. In case of MR, for examp= le, tasks talk to the AM and report their individual progress and counters = back to it, via a specific protocol (over the network), giving the AM more = data to compute an near-accurate global progress. >>=20 >> On Fri, Sep 20, 2013 at 12:18 AM, John Lilley = wrote: >>> How does a YARN application master typically query ongoing status=20 >>> (like percentage completion) of its tasks? >>>=20 >>> I would like to be able to ultimately relay information to the user lik= e: >>>=20 >>> 100 tasks are scheduled >>>=20 >>> 10 tasks are complete >>>=20 >>> 4 tasks are running and they are (4%, 10%, 50%, 70%) complete >>>=20 >>> But, given that YARN tasks are simply executables, how can the AM=20 >>> even get at this information? Can the AM get access to stdout/stderr? >>>=20 >>> Thanks >>>=20 >>> John >>>=20 >>>=20 >>=20 >>=20 >>=20 >> -- >> Harsh J >=20 >=20 >=20 > -- > Harsh J -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ --=20 CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to= =20 which it is addressed and may contain information that is confidential,=20 privileged and exempt from disclosure under applicable law. If the reader= =20 of this message is not the intended recipient, you are hereby notified that= =20 any printing, copying, dissemination, distribution, disclosure or=20 forwarding of this communication is strictly prohibited. If you have=20 received this communication in error, please contact the sender immediately= =20 and delete it from your system. Thank You. --Apple-Mail=_8724AAF5-E9B4-4E8A-96AB-DEF27CD1ED5E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1 Yep, typically, the AM sh= ould pass it's host:port to the task as part of either the cmd-line for the= task or in it's env. That is what is done by MR AM.

hth= ,
Arun

On Sep 21, 2013, at 6:52 AM, John = Lilley <john.lilley@redpoint= .net> wrote:

Thanks Harsh!  The data-transport format is pretty eas= y, but how is the RPC typically set up?  Does the AM open a listen por= t to accept the RPC from the tasks, and then pass the port/URI to the tasks= when they are spawned as command-line or environment?
john

-----= Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, September 20, 2013 7:47 AMTo: <user@hadoop.apache.org>
Subject: Re: Task status query

Right now its MR specific (= TaskUmbilicalProtocol) - YARN doesn't have any reusable items here yet, but= there are easy to use RPC libs such as Avro and Thrift out there that make= it easy to do such things once you define what you want in a schema/spec f= orm.

On Fri, Sep 20, 2013 at 5:32 PM, John Lilley <john.lilley@redpoint.net> wrote:
Thanks Harsh.  Is this protocol something that= is available to all AMs/tasks?  Or is it up to each AM/task pair to d= evelop their own protocol?
john

-----Original Message-----
Fro= m: Harsh J [mailto:harsh@cloudera.com]<= br>Sent: Thursday, September 19, 2013 9:20 PM
To: <user@hadoop.apache.org>
Subject: Re: Task= status query

Hi John,

YARN tasks can be more than simple exe= cutables. In case of MR, for example, tasks talk to the AM and report their= individual progress and counters back to it, via a specific protocol (over= the network), giving the AM more data to compute an near-accurate global p= rogress.

On Fri, Sep 20, 2013 at 12:18 AM, John Lilley <john.lilley@redpoint.net> wrote= :
How does a YARN application master typically= query ongoing status
(like percentage completion) of its tasks?
I would like to be able to ultimately relay information to the user like:<= br>
100 tasks are scheduled

10 tasks are complete

4 tasks = are running and they are (4%, 10%, 50%, 70%) complete

But, given tha= t YARN tasks are simply executables, how can the AM
even get at this in= formation?  Can the AM get access to stdout/stderr?

Thanks
<= br>John





--
Harsh J
<= br>

--
Harsh J

--
Arun C. Murthy


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --Apple-Mail=_8724AAF5-E9B4-4E8A-96AB-DEF27CD1ED5E--