hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <cdoug...@apache.org>
Subject Re: Yarn NodeManager services
Date Tue, 11 Oct 2011 21:33:25 GMT
On Tue, Oct 11, 2011 at 2:05 PM,  <Milind.Bhandarkar@emc.com> wrote:
> As part of MPI implementation in Yarn (aka Hamster), I was looking at
> refactoring some of the functionality into node manager services, so that
> it can be reused by other frameworks too. (Based on the discussion with
> some folks led me to believe that suffle etc is also being planned as NM
> service, separating it from deep integration with node manager.)

A general-purpose transfer protocol for intermediate data was designed
and scrapped for time. So was a local service allocated and shared by
resident containers. Instead, the NM supports loading modules (like
ShuffleHandler) at startup that implement auxiliary (or its
misspelling; my bad), ops-approved services, particularly shared
protocols independent of container and application allocation.

> I could not find the documentation for building NM services, in the MR-279
> architecture PDF. Does the service need to reside inside the NM jvm ?


>  How
> does NM talk to the service ?

The NM loads and starts a list of services from the config by
classname. Each of these services follows the same lifecycle as the
other NM components. Each service is given a name in the config, which
is also its address for containers launched on that machine. Each
container can pass an opaque blob to the service to initialize it with
container-specific data. For the MR shuffle, these are the job

Don't bother listing everything that's wrong with this model. Its
principal virtue was speed of implementation.

> How do containers discover the service ?

They don't. They need to know the name of the service a priori. -C

> (I
> know I can look at the code, but if there is already some docs (or even
> philosophical musings :-), it will make my life easier.
> Thanks,
> - milind
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)

View raw message