hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4149) JobQueueJobInProgressListener.jobUpdated() might not work as expected
Date Tue, 07 Oct 2008 14:29:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637502#action_12637502
] 

Hemanth Yamijala commented on HADOOP-4149:
------------------------------------------

This definitely seems to work. My only worry is that it will depend very heavily on the fact
that we have the old job status. If there is any case where this will not be available, the
solution breaks. For e.g. in the default scheduler, the {{JobQueueJobInProgressListener}}
removes the object from the queue even in {{jobRemoved}} which will not have the old job status.


 

> JobQueueJobInProgressListener.jobUpdated() might not work as expected
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-4149
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4149
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.19.0
>
>
> {{JobQueueJobInProgressListener}} uses a {{TreeSet}} to store the sorted collection of
{{JobInProgress}} objects. The comparator used to sort the JIPs follow the following order
> - priority (>=)
> - start time (<=)
> - job id [jt-identifier, job-index] (<=)
> If any JIP object is changed w.r.t priority or start-time, then the TreeSet will be inconsistent.
Hence doing  a delete might not work. Consider the following
> 1) jobs are submitted in the following order 
> ||number||jobid||priority||
> |1|j1|NORMAL|
> |2|j2|LOW|
> |3|j3|NORMAL|
> 2) The sorted collection will be in the order : {{j1,j3,j2}}
> 3) If job3's priority is changed to LOW then the collection wont change but delete will
bail out on j1 itself as the comparator will return a -ve number. TreeSet uses the comparator
both for sorting and deleting. If  i indicates the index in the collection and obj represents
the object under consideration, then looks like TreeSet.remove(obj) follows something like
 :
> - continue to search if the compare(i, obj) is -ve
> - bail out if the compare(i, obj) is +ve
> - delete the obj of compare(i,obj) == 0

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message