hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: yarn-site.xml and aux-services
Date Wed, 05 Jun 2013 15:34:54 GMT
John,

The format is ID and sub-config based:

First, you define an ID as a service, like the string "foo". This is
the ID the applications may lookup in their container responses map we
discussed over another thread (around shuffle handler).

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>foo</value>
</property>

Then you define an actual implementation class for that ID "foo", like so:

<property>
<name>yarn.nodemanager.aux-services.foo.class</name>
<value>com.mypack.MyAuxServiceClassForFoo</value>
</property>

If you have multiple services foo and bar, then it would appear like
the below (comma separated IDs and individual configs):

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>foo,bar</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.foo.class</name>
    <value>com.mypack.MyAuxServiceClassForFoo</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.bar.class</name>
    <value>com.mypack.MyAuxServiceClassForBar</value>
</property>

On Wed, Jun 5, 2013 at 8:42 PM, John Lilley <john.lilley@redpoint.net> wrote:
> Good, I was hoping that would be the case.  But what are the mechanics of it?  Do I just
add another entry?  And what exactly is "madreduce.shuffle"?  A scoped class name?  Or a key
string into some map elsewhere?
>
> e.g. like:
>
> <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>mapreduce.shuffle</value>
> </property>
> <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>myauxserviceclassname</value>
> </property>
>
> Concerning auxiliary services -- do they communicate with NodeManager via RPC?  Is there
an interface to implement?  How are they opened and closed with NodeManager?
>
> Thanks
> John
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Tuesday, June 04, 2013 11:58 PM
> To: <user@hadoop.apache.org>
> Subject: Re: yarn-site.xml and aux-services
>
> Yes, thats what this is for. You can implement, pass in and use your own AuxService.
It needs to be on the NodeManager CLASSPATH to run (and NM has to be restarted to apply).
>
> On Wed, Jun 5, 2013 at 4:00 AM, John Lilley <john.lilley@redpoint.net> wrote:
>> I notice the yarn-site.xml
>>
>>
>>
>>   <property>
>>
>>     <name>yarn.nodemanager.aux-services</name>
>>
>>     <value>mapreduce.shuffle</value>
>>
>>     <description>shuffle service that needs to be set for Map Reduce
>> to run </description>
>>
>>   </property>
>>
>>
>>
>> Is this a general-purpose hook?
>>
>> Can I tell yarn to run *my* per-node service?
>>
>> Is there some other way (within the recommended Hadoop framework) to
>> run a per-node service that exists during the lifetime of the NodeManager?
>>
>>
>>
>> John Lilley
>>
>> Chief Architect, RedPoint Global Inc.
>>
>> 1515 Walnut Street | Suite 200 | Boulder, CO 80302
>>
>> T: +1 303 541 1516  | M: +1 720 938 5761 | F: +1 781-705-2077
>>
>> Skype: jlilley.redpoint | john.lilley@redpoint.net | www.redpoint.net
>>
>>
>
>
>
> --
> Harsh J



-- 
Harsh J

Mime
View raw message