hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laxman Ch <laxman....@gmail.com>
Subject Re: Max Parallel task executors
Date Fri, 06 Nov 2015 12:45:26 GMT
Hi Sandeep,

Please configure the following items to the cores and memory per node you
wanted to allocate for Yarn containers.
Their defaults are 8 cores and 8GB. So that's the reason you were stuck at
31 (4nodes * 8cores - 1 AppMaster)

http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
yarn.nodemanager.resource.cpu-vcores
yarn.nodemanager.resource.memory-mb


On 6 November 2015 at 17:59, sandeep das <yarnhadoop@gmail.com> wrote:

> May be to naive to ask but How do I check that?
> Sometimes there are almost 200 map tasks pending to run but at a time only
> 31 runs.
>
> On Fri, Nov 6, 2015 at 5:57 PM, Chris Mawata <chris.mawata@gmail.com>
> wrote:
>
>> Also check that you have more than 31 blocks to process.
>> On Nov 6, 2015 6:54 AM, "sandeep das" <yarnhadoop@gmail.com> wrote:
>>
>>> Hi Varun,
>>>
>>> I tried to increase this parameter but it did not increase number of
>>> parallel tasks but if It is decreased then YARN reduces number of parallel
>>> tasks. I'm bit puzzled why its not increasing more than 31 tasks even after
>>> its value is increased.
>>>
>>> Is there any other configuration as well which controls on how many
>>> maximum tasks can execute in parallel?
>>>
>>> Regards,
>>> Sandeep
>>>
>>> On Tue, Nov 3, 2015 at 7:29 PM, Varun Vasudev <vvasudev@apache.org>
>>> wrote:
>>>
>>>> The number of parallel tasks that are run depends on the amount of
>>>> memory and vcores on your machines and the amount of memory and vcores
>>>> required by your mappers and reducers. The amount of memory can be set
>>>> via yarn.nodemanager.resource.memory-mb(the default is 8G). The amount of
>>>> vcores can be set via yarn.nodemanager.resource.cpu-vcores(the default
>>>> is 8 vcores).
>>>>
>>>> -Varun
>>>>
>>>> From: sandeep das <yarnhadoop@gmail.com>
>>>> Reply-To: <user@hadoop.apache.org>
>>>> Date: Monday, November 2, 2015 at 3:56 PM
>>>> To: <user@hadoop.apache.org>
>>>> Subject: Max Parallel task executors
>>>>
>>>> Hi Team,
>>>>
>>>> I've a cloudera cluster of 4 nodes. Whenever i submit a job my only 31
>>>> parallel tasks are executed whereas my machines have more CPU available but
>>>> still YARN/AM does not create more task.
>>>>
>>>> Is there any configuration which I can change to start more MAP/REDUCER
>>>> task in parallel?
>>>>
>>>> Each machine in my cluster has 24 CPUs.
>>>>
>>>> Regards,
>>>> Sandeep
>>>>
>>>
>>>
>


-- 
Thanks,
Laxman

Mime
View raw message