hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo yang <bobyan...@gmail.com>
Subject Re: Running YARN in-process Application Master?
Date Tue, 29 Apr 2014 15:29:07 GMT
This is great info for me. Thanks Oleg! I will take a look. Hope it can
also fit in our production environment.

Best Regards,
Bo


On Tue, Apr 29, 2014 at 3:38 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:

> Yes there is. You can provide your own implementation of
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor and
> configure it as  'yarn.nodemanager.container-executor.class' property.
> There you can bypass Shell and create your own way of invoking processes.
> Obviously it only makes sense for testing with mini-cluster during the
> development and I think that's what you meant.
> I've been playing with it lately, so feel free to take a look -
> https://github.com/olegz/yaya/wiki/CoreFeatures#in-jvm-container-executor.
> At least you'll get some ideas out of it.
>
> Cheers
> Oleg
>
>
> On Mon, Apr 28, 2014 at 3:57 PM, bo yang <bobyangbo@gmail.com> wrote:
>
>> Hi All,
>>
>> I just joined this group, and not sure whether this question was
>> discussed before.
>>
>> Is it possible to run Application Master within the same process as Node
>> Manager? If not, any plan to support it future?
>>
>> I am asking this because we might want to use YARN as a job dispatching
>> system. We have our own heavy-weight service which is not practical to
>> be launched as a new process for each new job. A possible thing we can do
>> is to use YARN to launch a new dummy Application Master process which will
>> dispatch the job to our service. But that new dummy process is a little
>> waste. So if YARN supports in-process Application Master, that will be
>> great.
>>
>> Thanks,
>> Bo
>>
>>
>

Mime
View raw message