hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo yang <bobyan...@gmail.com>
Subject Re: Any issue with large concurrency due to single active instance of YARN Resource Manager?
Date Tue, 02 Sep 2014 22:16:09 GMT
Hi Zhijie,

Looks great that YARN is mature enough to support so large cluster. It
gives me more confidence to use :)

Thanks a lot!

Best,
Bo



On Tue, Sep 2, 2014 at 2:41 PM, Zhijie Shen <zshen@hortonworks.com> wrote:

> Hi Bo,
>
> I don't have the exact number about the max concurrent job. FYI, RM is
> multip-threaded, but the threads are working for different purpose. For
> example, scheduler and rmstatestore has their separate thread, and RPC
> calls are on individual threads as well. It's complicated to evaluate the
> upper bound of concurrent apps, but I've heard of the YARN cluster
> deployment on a cluster of thousands of nodes.
>
> Thanks,
> Zhijie
>
>
> On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bobyangbo@gmail.com> wrote:
>
>> Hi Zhijie,
>>
>> That is great to know. Thanks!
>>
>> So there seems no be much limit to support large concurrency. To move
>> this question further, what might be the max number of concurrent jobs
>> which one Resource Manager could support? Is there any numbers from your
>> experience?
>>
>> Thanks,
>> Bo
>>
>>
>>
>>
>>
>>
>> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zshen@hortonworks.com>
>> wrote:
>>
>>> Hi Bo,
>>>
>>> RM doesn't create an individual thread for each running app. The app
>>> life cycle management is event driven. There's a dispatcher, which runs on
>>> one thread to handle the events for all apps.
>>>
>>> Zhijie
>>>
>>>
>>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bobyangbo@gmail.com> wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> I am thinking how many concurrent jobs a single Resource Manager might
>>>> be able to manage? Following is my understanding, please correct me if I
am
>>>> wrong.
>>>>
>>>> Let's say if we have 1000 concurrent jobs running. Resource Manager
>>>> will have 1000 records in memory to manage these jobs. And it will also
>>>> have 1000 threads, where each thread is waiting for one job to finish.
>>>>
>>>> The memory part will probably be ok. For the 1000 threads, will there
>>>> be any potential problem?
>>>>
>>>> Thanks,
>>>> Bo
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message