hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
Date Fri, 03 Nov 2017 02:14:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16236972#comment-16236972

Allen Wittenauer commented on HADOOP-13514:

Latest version of surefire is 2.20.1, released in September.

I've been spending a few weeks looking at our usage on the ASF build machines.  We typically
have anywhere from 8 to 40 zombie JVMs running around after the hadoop-common, hadoop-hdfs,
and hadoop-mapreduce allegedly complete. These do nothing but eat resources. As a result,
timeouts are pretty normal and problems that are claimed to be "environmental" are mostly
self-inflicted by badly written tests that really only get any stress on the build machines.

I'm fairly convinced that HDFS-12711 can be prevented with SUREFIRE-773, which was closed
as a dupe of SUREFIRE-524.  At this point, I think we absolutely need to make this a priority
to get into at least trunk and branch-2 ASAP.  

[It's easy to say... "this wasn't a problem before!"... I'm not so convinced it wasn't.  I
think we just didn't see it as often.  But new tests are doing new things that eat more resources
and stay around longer which impacts more runs.  Additionally, running unit tests under Docker
guaranteed that when the container died, so did all of these stale JVMs.  ]

> Upgrade maven surefire plugin to 2.19.1
> ---------------------------------------
>                 Key: HADOOP-13514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13514
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 2.8.0
>            Reporter: Ewan Higgs
>            Assignee: Akira Ajisaka
>            Priority: Major
>         Attachments: HADOOP-13514-addendum.01.patch, HADOOP-13514-testing.001.patch,
HADOOP-13514-testing.002.patch, HADOOP-13514-testing.003.patch, HADOOP-13514-testing.004.patch,
HADOOP-13514.002.patch, HADOOP-13514.003.patch, HADOOP-13514.004.patch, HADOOP-13514.005.patch,
> A lot of people working on Hadoop don't want to run all the tests when they develop;
only the bits they're working on. Surefire 2.19 introduced more useful test filters which
let us run a subset of the tests that brings the build time down from 'come back tomorrow'
to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true \"-Dtest=org.apache.hadoop.fs.*,
org.apache.hadoop.hdfs.*, org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command line but it
would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true \"-Dtest=org.apache.hadoop.fs.*,
org.apache.hadoop.hdfs.*, org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message