hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
Date Wed, 16 Aug 2017 23:07:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129582#comment-16129582

Allen Wittenauer commented on MAPREDUCE-4980:

bq. I wasn't able to reproduce the TestNNBench failure locally

It's extremely racy and seems to only show up under heavy load. Fun!  You might try increasing
the thread count.

bq. Looks like lots of things want to scribble on the same areas in target/test-dir.

The test-dir/x/dfs thing is a red herring.  The test dirs get re-used by different threads
at different times by assigning the various system properties when they are launched.  You'll
see the same sort of thing when running parallel in hadoop-hdfs; it's a product of the MiniDFSCluster.
 It's a little confusing and I sort of wish MiniDFSCluster always unique'd the directory (but
in a smarter way than MiniYARNCluster).  On the flip side, I'm too lazy to set the property
that gives each test it's own HDFS base dir. ;)

I'd need to see stack traces or something else to help diagnose further. If you see any "x
didn't cleanup in 20 secs" or whatever, those are the timing issues I'm still working out.
Finding a not-insane multiplier that works on Jenkins reliably has been hard. Suspect there
are a lot of dead processes on those machines. :(

BTW, there are a TON of expected and/or "useless" stack traces (I'm looking at you, metrics.)
in the test output.  The logging doesn't provide a lot of hints and so be careful about that

> Parallel test execution of hadoop-mapreduce-client-core
> -------------------------------------------------------
>                 Key: MAPREDUCE-4980
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
>             Project: Hadoop Map/Reduce
>          Issue Type: Test
>          Components: test
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Tsuyoshi Ozawa
>            Assignee: Andrey Klochkov
>         Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, MAPREDUCE-4980.012.patch,
MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch,
MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch,
> The maven surefire plugin supports parallel testing feature. By using it, the tests can
be run more faster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message