hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yehia Elshater <y.z.elsha...@gmail.com>
Subject Re: Fair Scheduler issue
Date Sat, 02 Aug 2014 19:13:06 GMT
Hi Julien,

Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
example (the other 3 for OS) ?

Thanks



On 1 August 2014 05:41, Julien Naour <julnaour@gmail.com> wrote:

> Hello,
>
> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>
> Recently we passed from 2 users to 8. We need now a more appropriate
> Scheduler.
> We begin with Capacity Scheduler. There was some issues with the different
> queues particularly when using some spark shell that used some resources
> for a long time.
> So we decide to try Fair Scheduler which seems to be a good solution.
> The problem is that FairScheduler doesn't allow all available resources.
> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
> 45% for 3 jobs. The problem could come from shells that take resources for
> a long time.
>
> We tried some configuration like
> yarn.scheduler.fair.user-as-default-queue=false
> or play with the minimum ressources allocated minResources in
> fair-scheduler.xml but it doesn't seems to resolve the issue.
>
> Any advices or good practices to held a good Fair Scheduler?
>
> Regards,
>
> Julien
>

Mime
View raw message