hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4428) Job Priorities are not handled properly
Date Tue, 21 Oct 2008 06:47:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641294#action_12641294
] 

Vivek Ratan commented on HADOOP-4428:
-------------------------------------

bq. We would need to prevent a huge number of jobs being setup at the same time, so we should
set a cap for how many jobs can be initialized at once. (3 jobs / user and 10 jobs for all
users?)

Agreed. I actually wanted to suggest something similar but forgot to add it to my comments.
Basically, you init jobs up to a limit, then you move on to the next queue. This patch should
be changed to do this. 

bq. we must keep the running jobs sorted by priority and submit time, just as the waiting
list is

Fair enough. I can see use cases for this. We should open another Jira for this. Implementing
this functionality (sorting of run queue) will also make this Jira's solution more fair, since
we won't rely on the completion times of setup tasks for jobs. 

Along the same vein, there are a few more suggestions I have, for which I'll open up separate
Jiras: separation of setup/clean task creation from initTasks(), and dealing with synchronous
calls to initTasks() which can take a long time (thanks, Hemanth, for bringing that up). 

> Job Priorities are not handled properly 
> ----------------------------------------
>
>                 Key: HADOOP-4428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4428
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>         Environment: Cluster:  106 TTs  MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159 
>            Reporter: Karam Singh
>            Assignee: Vinod K V
>            Priority: Blocker
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message