hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szilard Nemeth (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
Date Mon, 29 Oct 2018 12:42:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667138#comment-16667138
] 

Szilard Nemeth edited comment on YARN-5106 at 10/29/18 12:41 PM:
-----------------------------------------------------------------

About the test failures: 
 1. TestAMRMClient.testAMRMClientWithContainerResourceChange: seems to be unrelated (or flaky?)
 2. TestContainerAllocation: The same set of testcases are failed on trunk and with my patch
(local execution), so this is also unrelated.
 3. TestAllocationFileLoaderService.testQueueNameContainingNBWhitespace: Passes for me after
15 runs of the whole class. 
 Added a @Before method that deletes the allocation file, which is a good practice on its
own, regardless of the failure we experienced with the Jenkins results.

4. TestFairScheduler.testHierarchicalQueueAllocationFileParsing: Fixed the test, it was a
trivial one as my patch did not add queueC and queueD as a children of queueA, so it was due
to a difference in the structure of the allocation file.

5. TestFairScheduler.testFairShareAndWeightsInNestedUserQueueRule: This also generated changes
in the allocation.xml file compared to the original code, so I fixed it.
 However, the config my patch was generated with the testcases' code was invalid in a way
that it procuded an NPE.
 I created YARN-8951 for this to investigate further.

Notes for patch001:

I just did a simple find for all tests that generates allocation files (i.e. searched for
"<allocations>") and replaced all of queue file creations with {{PrintWriter}}s with
the queue config file builder introduced in YARN-7841.
 There's one exception, though: TestFairSchedulerPreemption#writeAllocFile, this is because
work is in progress on YARN-8059, so I wanted to avoid huge merge conflicts.
 If YARN-8059 goes in first, I will rebase the patch and apply the changes to preemption test,
otherwise I will create a separate jira to apply the changes to {{TestFairSchedulerPreemption}}
as well.


was (Author: snemeth):
About the test failures: 
1. TestAMRMClient.testAMRMClientWithContainerResourceChange: seems to be unrelated (or flaky?)
2. TestContainerAllocation: The same set of testcases are failed on trunk and with my patch
(local execution)
3. TestAllocationFileLoaderService.testQueueNameContainingNBWhitespace: Passes for me after
15 runs of the whole class. 
Added a @Before method that deletes the allocation file, which is a good practice on its own,
regardless of the failure we experienced from the Jenkins run.

4. TestFairScheduler.testHierarchicalQueueAllocationFileParsing: Fixed the test, it was a
trivial one as my patch did not add queueC and queueD as a children of queueA, so it was due
to a difference in the structure of the allocation file.

5. TestFairScheduler.testFairShareAndWeightsInNestedUserQueueRule: This also generated changes
in the allocation.xml file compared to the original code, so I fixed it.
However, the config my patch was generated with the testcases' code was invalid in a way that
it procuded an NPE.
I created YARN-8951 for this to investigate further.

Notes for patch001:

I just did a simple find for all tests that generates allocation files (i.e. searched for
"<allocations>") and replaced all of queue file creations with printwriters with the
queue config file builder introduced in YARN-7841.
There's one exception, though: TestFairSchedulerPreemption#writeAllocFile, this is because
work is in progress on YARN-8059, so I wanted to avoid huge merge conflicts.
If YARN-8059 goes in first, I will rebase the patch and apply the changes to preemption test,
otherwise I will create a separate jira to apply the changes to \{{TestFairSchedulerPreemption}}
as well.

> Provide a builder interface for FairScheduler allocations for use in tests
> --------------------------------------------------------------------------
>
>                 Key: YARN-5106
>                 URL: https://issues.apache.org/jira/browse/YARN-5106
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.8.0
>            Reporter: Karthik Kambatla
>            Assignee: Szilard Nemeth
>            Priority: Major
>              Labels: newbie++
>         Attachments: YARN-5106.001.patch
>
>
> Most, if not all, fair scheduler tests create an allocations XML file. Having a helper
class that potentially uses a builder would make the tests cleaner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message