hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11984) Enable parallel JUnit tests in pre-commit.
Date Sat, 16 May 2015 14:08:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546761#comment-14546761

Chris Nauroth commented on HADOOP-11984:

That was pretty good overall.  I see recent hadoop-hdfs test runs have been taking ~2.5 hours.
 This one was ~45 minutes.  There are a couple of things to follow up on before we can commit
# {{TestRenameWithSnapshots}} failed due to {{OutOfMemoryError}}.  I don't know yet if this
is a problem in the test code or if we really over-taxed the Jenkins host by running too many
JVM processes.
# Numerous tests were reported as timed out, but they actually passed.  The test-patch.sh
logic that checks for timed out tests works by looking for a "Running" line that is not followed
by a "Tests run" line.  This logic won't work with parallel execution, because the output
from multiple JUnit processes gets interleaved non-deterministically.  We'll need to figure
out a different way to do the timed out check.
# The parallel-tests profile is only implemented for hadoop-common and hadoop-hdfs, so other
sub-modules won't get the benefit yet.  MapReduce is tracked in MAPREDUCE-4980.  I don't think
we have a YARN jira.  We don't have to wait for these to get done.  Those modules just won't
get the benefit until they're done.

Next, I'm going to try another patch with a forced hadoop-common change to see how that does.

> Enable parallel JUnit tests in pre-commit.
> ------------------------------------------
>                 Key: HADOOP-11984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11984
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-11984.001.patch
> HADOOP-9287 and related issues implemented the parallel-tests Maven profile for running
JUnit tests in multiple concurrent processes.  This issue proposes to activate that profile
during pre-commit to speed up execution.

This message was sent by Atlassian JIRA

View raw message