hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11984) Enable parallel JUnit tests in pre-commit.
Date Sat, 16 May 2015 21:02:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546938#comment-14546938
] 

Chris Nauroth commented on HADOOP-11984:
----------------------------------------

We saw {{OutOfMemoryError}} in that hadoop-common run too.  We'll need to take a closer look
at how Surefire allocates heap size for multiple concurrent JUnit processes.

A few of the file system contract tests failed, and I think it's because they collided on
the {{test.build.data}} directory.  In hadoop-hdfs, we made a pom.xml change to parameterize
that directory based on the JUnit process number.  That isn't done in the hadoop-common pom.xml,
so we'll need to change it.

{code}
              <systemPropertyVariables>
                <test.build.data>${test.build.data}/${surefire.forkNumber}</test.build.data>
                <hadoop.tmp.dir>${hadoop.tmp.dir}/${surefire.forkNumber}</hadoop.tmp.dir>
              </systemPropertyVariables>
{code}

I'm not sure about {{TestZKFailoverControllerStress}} yet, although that one has been flaky
in the past.

bq. Does it make sense to move some of these work towards a Python script so that it is possible
to leverage more libraries?

I didn't have any particular solution in mind, so I think anything is on the table at this
point.  If we can find another heuristic for identifying timed out tests using bash/grep/awk,
then I think that's fine.  If we can't find a reliable heuristic, then parsing the XML sounds
good.

> Enable parallel JUnit tests in pre-commit.
> ------------------------------------------
>
>                 Key: HADOOP-11984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11984
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-11984.001.patch, HADOOP-11984.002.patch
>
>
> HADOOP-9287 and related issues implemented the parallel-tests Maven profile for running
JUnit tests in multiple concurrent processes.  This issue proposes to activate that profile
during pre-commit to speed up execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message