hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14696) parallel tests don't work for Windows
Date Thu, 03 Aug 2017 16:26:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113031#comment-16113031
] 

Allen Wittenauer commented on HADOOP-14696:
-------------------------------------------


bq. could we just define some property for the path, e.g $separator , then use that in the
property defs?
bq. there's always the ant <mkdir> command.

Unfortunately, it's not that simple. We need to be able to create a set of directories per
thread.  The thread count is defined at runtime.  This means a loop.  That eliminated <mkdir>;
I couldn't figure out how to loop other than the method that [~cnauroth] used when they wrote
the original antrun+JavaScript code.  The loop (obviously) needs to take input from maven
properties.  These properties are calculated based upon standard Maven ones (which are built
way before it even reads the pom.xml). Maven itself stores as full Windows paths.  So we're
popping these maven properties into the antrun JavaScript. 

The problem is that JavaScript (correctly?)  interprets Windows backslashes as escapes.  So
instead of C:\Tools\Source it gets turned into C:ToolsSource.  Now it's possible to switch
languages (Groovy, JRuby, Jython, etc).  This brings about three new problems:
* Will they handle the path problems on their own?  How will they deal with given a path that
looks like C:\Source\hadoop/target ?
* It adds yet more downloaded dependencies into the build.
* Do we really want to add Yet Another Language to the build system?

I opted for the devil we know and addded the new converted properties in a way that they are
available in all descendant poms.  As we get more modules running in parallel (I'm working
on rebasing MAPREDUCE-6674), they'll be able to use the same converted properties.  

bq. as the AWS patch policy is "always declare the endpoint you've tested against", which
s3 endpoint have you tested with?

I didn't.  The parallel-tests code is already present in the hadoop-aws pom.xml. If hadoop-aws
unit tests don't work in parallel on Linux now, then that profile shouldn't be there. 

> parallel tests don't work for Windows
> -------------------------------------
>
>                 Key: HADOOP-14696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14696
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 3.0.0-beta1
>         Environment: Windows
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Minor
>         Attachments: HADOOP-14696.00.patch, HADOOP-14696.01.patch
>
>
> If hadoop-common-project/hadoop-common is run with the -Pparallel-tests flag, it fails
in create-parallel-tests-dirs from the pom.xml
> {code}
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-parallel-tests-dirs)
on project hadoop-common: An Ant BuildException has occured: Directory F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\jenkinsjenkins-slaveworkspacehadoop-trunk-winshadoop-common-projecthadoop-common
arget\test\data\1 creation was not successful for an unknown reason
> [ERROR] around Ant part ...<script language="javascript">var baseDirs = [... @
4:33 in F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\target\antrun\build-main.xml
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message