Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of bobyangbo@gmail.com designates
 74.125.82.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+POG06TtXhmr_2g9sgavGxS6wX4sOo_VLBUKWDaXjZYnduH8g@mail.gmail.com>
References: 
 <CAK=ti8OGwr1n56=PPCWMBHNAwN=VHR11QvRd1WNd-ivfm4U03g@mail.gmail.com>
	<CA+POG06TtXhmr_2g9sgavGxS6wX4sOo_VLBUKWDaXjZYnduH8g@mail.gmail.com>
Date: Tue, 29 Apr 2014 08:29:07 -0700
Message-ID: 
 <CAK=ti8N0YsVXhhE3quTMvkON8f6f=BzyCBDZi8ba5bSTJVSX9w@mail.gmail.com>
Subject: Re: Running YARN in-process Application Master?
From: bo yang <bobyangbo@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d04426c46bca2af04f8301708

--f46d04426c46bca2af04f8301708
Content-Type: text/plain; charset=UTF-8

This is great info for me. Thanks Oleg! I will take a look. Hope it can
also fit in our production environment.

Best Regards,
Bo


On Tue, Apr 29, 2014 at 3:38 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:

> Yes there is. You can provide your own implementation of
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor and
> configure it as  'yarn.nodemanager.container-executor.class' property.
> There you can bypass Shell and create your own way of invoking processes.
> Obviously it only makes sense for testing with mini-cluster during the
> development and I think that's what you meant.
> I've been playing with it lately, so feel free to take a look -
> https://github.com/olegz/yaya/wiki/CoreFeatures#in-jvm-container-executor.
> At least you'll get some ideas out of it.
>
> Cheers
> Oleg
>
>
> On Mon, Apr 28, 2014 at 3:57 PM, bo yang <bobyangbo@gmail.com> wrote:
>
>> Hi All,
>>
>> I just joined this group, and not sure whether this question was
>> discussed before.
>>
>> Is it possible to run Application Master within the same process as Node
>> Manager? If not, any plan to support it future?
>>
>> I am asking this because we might want to use YARN as a job dispatching
>> system. We have our own heavy-weight service which is not practical to
>> be launched as a new process for each new job. A possible thing we can do
>> is to use YARN to launch a new dummy Application Master process which will
>> dispatch the job to our service. But that new dummy process is a little
>> waste. So if YARN supports in-process Application Master, that will be
>> great.
>>
>> Thanks,
>> Bo
>>
>>
>

--f46d04426c46bca2af04f8301708
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>This is great info for me. Thanks Oleg! I will take a=
 look. Hope it can also fit=C2=A0in our production environment.</div><div><=
span></span>=C2=A0</div><div>Best Regards,</div><div><span>Bo</span></div><=
/div><div class=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On Tue, Apr 29, 2014 at 3:38 AM, Oleg Zh=
urakousky <span dir=3D"ltr">&lt;<a href=3D"mailto:oleg.zhurakousky@gmail.co=
m" target=3D"_blank">oleg.zhurakousky@gmail.com</a>&gt;</span> wrote:<br><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex">
<div dir=3D"ltr">Yes there is. You can provide your own implementation of o=
rg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor and confi=
gure it as=C2=A0=C2=A0&#39;yarn.nodemanager.container-executor.class&#39; p=
roperty.<div>

There you can bypass Shell and create your own way of invoking processes. O=
bviously it only makes sense for testing with mini-cluster during the devel=
opment and I think that&#39;s what you meant.</div><div>I&#39;ve been playi=
ng with it lately, so feel free to take a look -=C2=A0<a href=3D"https://gi=
thub.com/olegz/yaya/wiki/CoreFeatures#in-jvm-container-executor" target=3D"=
_blank">https://github.com/olegz/yaya/wiki/CoreFeatures#in-jvm-container-ex=
ecutor</a>. At least you&#39;ll get some ideas out of it.</div>

<div><br></div><div>Cheers</div><span class=3D"HOEnZb"><font color=3D"#8888=
88"><div>Oleg</div></font></span></div><div class=3D"HOEnZb"><div class=3D"=
h5"><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Mon, A=
pr 28, 2014 at 3:57 PM, bo yang <span dir=3D"ltr">&lt;<a href=3D"mailto:bob=
yangbo@gmail.com" target=3D"_blank">bobyangbo@gmail.com</a>&gt;</span> wrot=
e:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;padding=
-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-l=
eft-style:solid"><div dir=3D"ltr"><div>Hi All,</div><div><br></div><div>I j=
ust joined this group, and not sure whether this question was discussed bef=
ore.</div>

<div><br></div><div>Is it possible to run Application Master within the sam=
e process as Node Manager? If not, any plan to support it future?</div>
<div><br></div><div>I am asking this because we might want to use YARN as a=
 job dispatching system. We have our own heavy-weight service which is not =
practical to be=C2=A0launched as=C2=A0a new process for each new job. A pos=
sible thing we can do is to use YARN to launch a new dummy Application Mast=
er process which will dispatch the job to our service. But that new dummy p=
rocess is a little waste. So if YARN supports in-process Application Master=
, that will be great.</div>


<div><br></div><div>Thanks,</div><div>Bo</div><div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--f46d04426c46bca2af04f8301708--