hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajith S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
Date Thu, 13 Aug 2015 03:52:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694655#comment-14694655
] 

Ajith S commented on YARN-3689:
-------------------------------

Hi [~lyn]

Thanks for reporting the issue. *org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FifoPolicy.FifoComparator.compare(Schedulable,
Schedulable)* seems fine to me as the lower the priority number, the schedule holds higher
value. But i think the problem is in Collection.sort in FSLeafQueue which will order the elements
in ascending order. Any thoughts.?

Here is a small test case
{code}

    //create 2 schedule with priority
    Schedulable s1 = createSchedulable(1);
    Schedulable s2 = createSchedulable(2);
    
    FifoPolicy fp = new FifoPolicy();
    Comparator<Schedulable> fpComparator = fp.getComparator();
    int compare = fpComparator.compare(s1, s2);
    //ensure smaller the number, is considered higher
    assertTrue(compare > 0);
    
    List<Schedulable> list = new ArrayList<>();
    list.add(s1);
    list.add(s2);
    Collections.sort(list,fpComparator);
    assertEquals(s1,list.get(0)); //fails
    assertEquals(s2,list.get(1)); //fails
  
{code}

> FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1"
and "s2" should change position when compare priority 
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3689
>                 URL: https://issues.apache.org/jira/browse/YARN-3689
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler, scheduler
>    Affects Versions: 2.5.0
>            Reporter: zhoulinlin
>            Assignee: Ajith S
>
> In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position
when compare priority.
> I did a test. Configured the schedulerpolicy "fifo",  submitted 2 jobs to the same queue.
> The result is below:
> 2015-05-20 11:57:41,449 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
before sort ------------------  
> 2015-05-20 11:57:41,449 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432094103221_0001 appPririty:4  appStartTime:1432094170038
> 2015-05-20 11:57:41,449 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432094103221_0002 appPririty:2  appStartTime:1432094173131
> 2015-05-20 11:57:41,449 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
after sort %%%%%%%%%%%%% 
> 2015-05-20 11:57:41,449 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432094103221_0001 appPririty:4  appStartTime:1432094170038  
> 2015-05-20 11:57:41,449 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432094103221_0002 appPririty:2  appStartTime:1432094173131  
> But when change the "s1" and "s2" position like below:
> public int compare(Schedulable s1, Schedulable s2) {
>       int res = s2.getPriority().compareTo(s1.getPriority());
> .........}
> The result:
> 2015-05-20 11:36:37,119 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
before sort ------------------ 
> 2015-05-20 11:36:37,119 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432090734333_0009 appPririty:4  appStartTime:1432092992503
> 2015-05-20 11:36:37,119 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432090734333_0010 appPririty:2  appStartTime:1432092996437
> 2015-05-20 11:36:37,119 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
after sort %%%%%%%%%%%%% 
> 2015-05-20 11:36:37,119 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432090734333_0010 appPririty:2  appStartTime:1432092996437
> 2015-05-20 11:36:37,119 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
appName:application_1432090734333_0009 appPririty:4  appStartTime:1432092992503 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message