hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: Fair scheduler fairness question
Date Wed, 10 Mar 2010 16:06:52 GMT
On 3/10/10 7:38 AM, "Neo Anderson" <javadeveloper999@yahoo.co.uk> wrote:

> I am learning how fair scheduler manage the jobs to allow each job share
> resource over time; but don't know if my understanding is correct or not.
> My scenario is that I have 3 data nodes and the cluster is configured using
> fair scheduler with three pools launched (e.g. A, B, C). Each pool is
> configured with '<maxRunningJobs>1</maxRunningJobs>.' Now the clients try
> submit 4 jobs (e.g. submitjob()) to 3 differt pools. For instance,
> the first job is submitted to pool A
> the second job is submitted to pool B
> the third job is submitted to pool B
> the fourth job is submitted to pool C
> So I expect that the first 3 jobs will occupy the free slots (the slots should
> be fool now.) Then the fourth job is submitted. But since the slots are full,
> and the fourth job should also have a slot executing its job; therefore, the
> third job will be terminated (or kill) so that the fourth job can be launched.
> Is my scenario correct?
> And if I am right, is there any key word searchable in the log to observe such
> activites (the job that is being killed e.g. the third job)?

A lot of it depends upon timing.  If there is a long enough pause between
job 1 and job 2, job 1 will take every slot available to it.  As job 1's
slots finish, job 2 and 4 would get those slots.   As job 2 finishes, job 3
will get its slots.

Slots are only freed by force if the scheduler you are using has
pre-emption.  I think some versions of fair share may have it.  Entire jobs
are never killed.

View raw message