hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Goeke <goeke.matt...@gmail.com>
Subject Fair Scheduler question: Fair share and its effect on max capacity
Date Thu, 08 Nov 2012 23:12:36 GMT
Pretty straight forward question but can the fair share factor actually
impact the total number of jobs / slots a pool can take up even if it is
the only pool with active jobs submitted?

We currently have a pool that has this configuration:
"minMaps": 2,

"minReduces": 1,
"maxMaps": 200,
"maxReduces": 66,
"maxRunningJobs": 200,
"minSharePreemptionTimeout": 300,
"weight": "4.0"

The total cluster capacity is over above 250 mappers but we are
finding that this pool is never able to hit that max capacity for maps
OR jobs even during load tests. I was about to bump the minMaps
property but I wanted to confirm that this could potentially help
alleviate our issue.


View raw message