hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4658) User limit is not expanding back properly.
Date Fri, 14 Nov 2008 11:56:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647593#action_12647593

Vivek Ratan commented on HADOOP-4658:

There are a couple of things going on here. 

The scheduler is notified by the JT when a job is considered 'done' (see HADOOP-4053). At
that point, we recompute the number of users who have submitted jobs in the queue. If this
notification is coming in late, you need to see why. But the scheduler's behavior is right
in terms of computing the user limit. The scheduler doesn't decide when a job is 'done'. The
JT does. And that, IMO,is correct behavior. 

Secondly, if we walk through the entire queue and no job can accept a slot (likely because,
as in this case, either jobs are over capacity or don't have tasks to run) and there are no
waiting tasks, we still want the slot to be used by the queue, so we walk through the queue
again, this tiem without considering user limits. Likely, the first job will start getting
a bunch of slots. This is by design, as it's really hard to argue what is fair in this case.
Taking one of your examples, suppose the queue capacity is 100 and we have four jobs from
four different users. Each is using 25 slots. J3 starts finishing up, and at some point, is
only running, say, 5 tasks. Also assume there are no waiting jobs. Now, what's the right behavior?
WHo should get the slot? J1, right? Who gets the next slot? You can argue that you want to
redistribute J3's 20 unused slots among J1, J2, and J4, but this recomputation gets really
complicated. So, we took a simpler approach by saying that free slots are offered to jobs
in order. So J1 will get a bunch of free slots, which will let it finish fast. Eventually
J3 finishes, is taken out of the queue, and user limits are recomputed. We couldn't think
of a simple and more fair approach here. Note that this situation is a bit rare. On a regular,
well-utilized cluster, you'll have a bunch of waiting jobs and they will start running. J3's
user's first waiting job will start running, which is the right thing. 

So, in summary, I'd say this is expected behavior. 

As for your comment on speculative tasks being run by J1 - that's really a different call.
J1 runs a speculative task if it legitimately has a speculate task to run, not because there's
a slot free. J1 can come back and say it doesn't have any task to run, in which case J2 is
looked at next. If J1 is running 17 speculative tasks, they're as genuine, and higher priority,
than J2's tasks, so I'd say that's still the right behavior. 

> User limit is not expanding back properly.
> ------------------------------------------
>                 Key: HADOOP-4658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4658
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>         Environment: GC=100% nodes=104, map_capacity=208, reduce_capacity=208, user-limit=25%;
>            Reporter: Karam Singh
>            Assignee: Amar Kamat
> User limit is not expanding back properly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message