hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11984) Enable parallel JUnit tests in pre-commit.
Date Tue, 19 May 2015 01:28:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549616#comment-14549616

Allen Wittenauer commented on HADOOP-11984:

bq.  Is it worthwhile to trade the additional complexity and possibly reduced robustness for
the optimized execution time?

I don't find the current code complex at all compared to some of the gymnastics happening
in hadoop-functions.sh.   I might have a different opinion if the docker bits ever get committed.

bq. The UNIX world supports bash pretty well. However, Hadoop officially supports Windows
as well. How does the compatibility story work for Windows folks? Unfortunately things like
path separators are quite annoying to handle.

We've already done it for the client code in trunk.  It's likely better than the DOS batch
code. My hunch is that it won't be that bad to get test-patch.sh running under cygwin.  More
complex than the client code though.

bq. What I'm trying to explore here from a maintainability / productivity prospective whether
we should allow writing supplement scripts in languages that have a wide range of libraries
to take care of things like text processing / OS manipulation, regardless it's Python, Perl
or other languages.

Until the portability issue gets dealt with (which will happen as more time passes), the answer
here is likely no without potentially significantly increasing the burden of the installed
software set.

Of course, unless someone is actually going to do the work, this whole conversation is sort
of moot.  Traditionally, the non-java parts of Hadoop are regularly ignored until it impacts
someone's corporate goals.   I don't expect that to change any time soon.

bq.I can't repro the OutOfMemoryError problems after setting reuseForks to false. I don't
know if I'm just overloading these hosts by running multiple processes with a maximum 1 GB
heap. I don't have much visibility into these hosts. If anyone else out there knows how to
find host-level details, please let me know. Otherwise, I'll probably need to coordinate with

Hmm. I wonder if the fact that MAVEN_OPTS is set to "-Xmx3072m -XX:MaxPermSize=768m" prior
to test-patch.sh invocation in the HADOOP precommit is part of the problem here.  

This is where it'd be super useful to run precommit in the Docker container so we could see
if it was actually a maven version problem. :(

> Enable parallel JUnit tests in pre-commit.
> ------------------------------------------
>                 Key: HADOOP-11984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11984
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-11984.001.patch, HADOOP-11984.002.patch, HADOOP-11984.003.patch,
HADOOP-11984.004.patch, HADOOP-11984.005.patch
> HADOOP-9287 and related issues implemented the parallel-tests Maven profile for running
JUnit tests in multiple concurrent processes.  This issue proposes to activate that profile
during pre-commit to speed up execution.

This message was sent by Atlassian JIRA

View raw message