hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4428) Job Priorities are not handled properly
Date Mon, 20 Oct 2008 15:07:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641033#action_12641033
] 

Hemanth Yamijala commented on HADOOP-4428:
------------------------------------------

In an offline discussion with Vivek, we found that this patch is not correct, as it moves
to the job in the next queue after initializing a job in the waiting queue. In fact, there
are two deeper issues:

- Currently the scheduler, which calls {{initTasks}} can trigger a set up task to be run.
This does not seem correct, because just like clean up tasks, setup tasks should be done outside
the control of the scheduler. This should be addressed in the framework, possibly as part
of HADOOP-4421 ?
- The capacity scheduler does not check for the status of a job to be running before assigning
tasks from it when it initializes jobs in the waiting queue. The implicit assumption here
was that after a call to {{initTasks}}, the job would be runnable. However, changes to the
framework  to run setup tasks broke this. Even if setup tasks are done independently of the
scheduler, it is still possible that the scheduler is looking at a job whose setup task is
still not complete, and hence by definition is not runnable.

The first point is more of a design issue, and hence not a blocker for 0.19, as currently
nothing breaks because of the approach.

The second point must be addressed in the capacity scheduler, which should actually fix the
problem raised in the bug report as well. 

> Job Priorities are not handled properly 
> ----------------------------------------
>
>                 Key: HADOOP-4428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4428
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>         Environment: Cluster:  106 TTs  MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159 
>            Reporter: Karam Singh
>            Assignee: Vinod K V
>            Priority: Blocker
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message