Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of john.lilley@redpoint.net
 designates 206.225.164.219 as permitted sender)
From: John Lilley <john.lilley@redpoint.net>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: RE: Task status query
Thread-Topic: Task status query
Thread-Index: Ac61aIt4OP487tKeSxGEJv7m0NkVtgAgnwmAAAN+MtAAEmbgAAAjwWUQ
Date: Sat, 21 Sep 2013 13:52:44 +0000
Message-ID: 
 <869970D71E26D7498BDAC4E1CA92226B86D385A5@MBX021-E3-NJ-2.exch021.domain.local>
References: 
 <869970D71E26D7498BDAC4E1CA92226B86D363CB@MBX021-E3-NJ-2.exch021.domain.local>
 <CAOcnVr1Hc57dDEx=U7cweuG76rVVZ5_DGuKBL0njzRHOK_oZJA@mail.gmail.com>
 <869970D71E26D7498BDAC4E1CA92226B86D37282@MBX021-E3-NJ-2.exch021.domain.local>
 <CAOcnVr0PEGWqHcBc_Me9SgQDUwiC3ni+XbQuwzF+2MFNZG++bg@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr0PEGWqHcBc_Me9SgQDUwiC3ni+XbQuwzF+2MFNZG++bg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks Harsh!  The data-transport format is pretty easy, but how is the RPC=
 typically set up?  Does the AM open a listen port to accept the RPC from t=
he tasks, and then pass the port/URI to the tasks when they are spawned as =
command-line or environment?
john

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]=20
Sent: Friday, September 20, 2013 7:47 AM
To: <user@hadoop.apache.org>
Subject: Re: Task status query

Right now its MR specific (TaskUmbilicalProtocol) - YARN doesn't have any r=
eusable items here yet, but there are easy to use RPC libs such as Avro and=
 Thrift out there that make it easy to do such things once you define what =
you want in a schema/spec form.

On Fri, Sep 20, 2013 at 5:32 PM, John Lilley <john.lilley@redpoint.net> wro=
te:
> Thanks Harsh.  Is this protocol something that is available to all AMs/ta=
sks?  Or is it up to each AM/task pair to develop their own protocol?
> john
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Thursday, September 19, 2013 9:20 PM
> To: <user@hadoop.apache.org>
> Subject: Re: Task status query
>
> Hi John,
>
> YARN tasks can be more than simple executables. In case of MR, for exampl=
e, tasks talk to the AM and report their individual progress and counters b=
ack to it, via a specific protocol (over the network), giving the AM more d=
ata to compute an near-accurate global progress.
>
> On Fri, Sep 20, 2013 at 12:18 AM, John Lilley <john.lilley@redpoint.net> =
wrote:
>> How does a YARN application master typically query ongoing status=20
>> (like percentage completion) of its tasks?
>>
>> I would like to be able to ultimately relay information to the user like=
:
>>
>> 100 tasks are scheduled
>>
>> 10 tasks are complete
>>
>> 4 tasks are running and they are (4%, 10%, 50%, 70%) complete
>>
>> But, given that YARN tasks are simply executables, how can the AM=20
>> even get at this information?  Can the AM get access to stdout/stderr?
>>
>> Thanks
>>
>> John
>>
>>
>
>
>
> --
> Harsh J


--
Harsh J