hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahadev Konar <maha...@hortonworks.com>
Subject Re: Yarn NodeManager services
Date Tue, 11 Oct 2011 21:28:04 GMT
Shuffle is an auxiliary service in the NM. You might want to take a look at AuxServices.java
in the NodeManager codebase.

Also, you can add config parameters to 
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>
(config knob in NodeManagers)

to add more configurable services in the NodeManager.

Hope that helps.
mahadev


On Oct 11, 2011, at 2:05 PM, <Milind.Bhandarkar@emc.com> <Milind.Bhandarkar@emc.com>
wrote:

> Hi,
> 
> As part of MPI implementation in Yarn (aka Hamster), I was looking at
> refactoring some of the functionality into node manager services, so that
> it can be reused by other frameworks too. (Based on the discussion with
> some folks led me to believe that suffle etc is also being planned as NM
> service, separating it from deep integration with node manager.)
> 
> I could not find the documentation for building NM services, in the MR-279
> architecture PDF. Does the service need to reside inside the NM jvm ?  How
> does NM talk to the service ? How do containers discover the service ? (I
> know I can look at the code, but if there is already some docs (or even
> philosophical musings :-), it will make my life easier.
> 
> Thanks,
> 
> - milind
> 
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
> 
> 
> 


Mime
View raw message