hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo yang <bobyan...@gmail.com>
Subject Re: Any issue with large concurrency due to single active instance of YARN Resource Manager?
Date Tue, 02 Sep 2014 17:42:53 GMT
Hi Zhijie,

That is great to know. Thanks!

So there seems no be much limit to support large concurrency. To move this
question further, what might be the max number of concurrent jobs which one
Resource Manager could support? Is there any numbers from your experience?


On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zshen@hortonworks.com> wrote:

> Hi Bo,
> RM doesn't create an individual thread for each running app. The app life
> cycle management is event driven. There's a dispatcher, which runs on one
> thread to handle the events for all apps.
> Zhijie
> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bobyangbo@gmail.com> wrote:
>> Hi Guys,
>> I am thinking how many concurrent jobs a single Resource Manager might be
>> able to manage? Following is my understanding, please correct me if I am
>> wrong.
>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>> have 1000 records in memory to manage these jobs. And it will also have
>> 1000 threads, where each thread is waiting for one job to finish.
>> The memory part will probably be ok. For the 1000 threads, will there be
>> any potential problem?
>> Thanks,
>> Bo
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

View raw message