hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oleksandr Shevchenko (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-7139) TestShuffleProvider#testShuffleProviders fails since a static common container launch context does't recreate
Date Tue, 11 Sep 2018 09:01:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Oleksandr Shevchenko updated MAPREDUCE-7139:
    Attachment: screenshot-2.png

> TestShuffleProvider#testShuffleProviders fails since a static common container launch
context does't recreate
> -------------------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-7139
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7139
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: test
>            Reporter: Oleksandr Shevchenko
>            Assignee: Oleksandr Shevchenko
>            Priority: Minor
>         Attachments: MAPREDUCE-7139.001.patch, screenshot-1.png, screenshot-2.png
> TestShuffleProvider#testShuffleProviders fails with the following error:
> {noformat}
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.assertTrue(Assert.java:41)
> 	at org.junit.Assert.assertNotNull(Assert.java:621)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.TestShuffleProvider.testShuffleProviders(TestShuffleProvider.java:114)
> {noformat}
> This error occurs only when we run TestShuffleProvider#testShuffleProviders test after
some test which creates TaskAttemptImpl (For example after tests in TestMapReduceChildJVM).
> TestShuffleProvider#testShuffleProviders creates container launch context, set additionals
NM aux services and expects to get these services from container launch context after creating.
> But we have the following code in TaskAttemptImpl:
> {code}
> static ContainerLaunchContext createContainerLaunchContext(
>       Map<ApplicationAccessType, String> applicationACLs,
>       Configuration conf, Token<JobTokenIdentifier> jobToken, Task remoteTask,
>       final org.apache.hadoop.mapred.JobID oldJobId,
>       WrappedJvmID jvmID,
>       TaskAttemptListener taskAttemptListener,
>       Credentials credentials) {
>     synchronized (commonContainerSpecLock) {
>       if (commonContainerSpec == null) {
>         commonContainerSpec = createCommonContainerLaunchContext(
>             applicationACLs, conf, jobToken, oldJobId, credentials);
>       }
>     }
> {code}
> The property "commonContainerSpec" is static. As the result, we do not recreate common
container launch within JVM. This is normal for AM JVM since all containers should be run
with one context. But this leads to an error when we run unit tests in one JVM and expect
to recreate container launch context.
> We can add a setter for "commonContainerSpec" with "VisibleForTesting" annotation and
reset CLC in a test if we need it.
> Also, we can cleanup CLC after running tests in @After fixture which create CLC but we
have too many tests which use TaskAttemptImpl and do not need cleanup.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message