hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julien Naour <julna...@gmail.com>
Subject Fair Scheduler issue
Date Fri, 01 Aug 2014 09:41:05 GMT
Hello,

I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.

Recently we passed from 2 users to 8. We need now a more appropriate
Scheduler.
We begin with Capacity Scheduler. There was some issues with the different
queues particularly when using some spark shell that used some resources
for a long time.
So we decide to try Fair Scheduler which seems to be a good solution.
The problem is that FairScheduler doesn't allow all available resources.
It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
45% for 3 jobs. The problem could come from shells that take resources for
a long time.

We tried some configuration like
yarn.scheduler.fair.user-as-default-queue=false
or play with the minimum ressources allocated minResources in
fair-scheduler.xml but it doesn't seems to resolve the issue.

Any advices or good practices to held a good Fair Scheduler?

Regards,

Julien

Mime
View raw message