hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Milind.Bhandar...@emc.com>
Subject Re: Yarn NodeManager services
Date Tue, 11 Oct 2011 21:43:44 GMT
This is cool. Exactly what I was looking for ! Thanks Chris & Mahadev.

- Milind

On 10/11/11 2:33 PM, "Chris Douglas" <cdouglas@apache.org> wrote:

>On Tue, Oct 11, 2011 at 2:05 PM,  <Milind.Bhandarkar@emc.com> wrote:
>> As part of MPI implementation in Yarn (aka Hamster), I was looking at
>> refactoring some of the functionality into node manager services, so
>> it can be reused by other frameworks too. (Based on the discussion with
>> some folks led me to believe that suffle etc is also being planned as NM
>> service, separating it from deep integration with node manager.)
>A general-purpose transfer protocol for intermediate data was designed
>and scrapped for time. So was a local service allocated and shared by
>resident containers. Instead, the NM supports loading modules (like
>ShuffleHandler) at startup that implement auxiliary (or its
>misspelling; my bad), ops-approved services, particularly shared
>protocols independent of container and application allocation.
>> I could not find the documentation for building NM services, in the
>> architecture PDF. Does the service need to reside inside the NM jvm ?
>>  How
>> does NM talk to the service ?
>The NM loads and starts a list of services from the config by
>classname. Each of these services follows the same lifecycle as the
>other NM components. Each service is given a name in the config, which
>is also its address for containers launched on that machine. Each
>container can pass an opaque blob to the service to initialize it with
>container-specific data. For the MR shuffle, these are the job
>Don't bother listing everything that's wrong with this model. Its
>principal virtue was speed of implementation.
>> How do containers discover the service ?
>They don't. They need to know the name of the service a priori. -C
>> (I
>> know I can look at the code, but if there is already some docs (or even
>> philosophical musings :-), it will make my life easier.
>> Thanks,
>> - milind
>> ---
>> Milind Bhandarkar
>> Greenplum Labs, EMC
>> (Disclaimer: Opinions expressed in this email are those of the author,
>> do not necessarily represent the views of any organization, past or
>> present, the author might be affiliated with.)

View raw message