hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9290) Some tests cannot load native library
Date Tue, 23 Apr 2013 19:23:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639448#comment-13639448

Chris Nauroth commented on HADOOP-9290:

We've been trying to find a graceful solution to this problem that involves defining the path
to libhadoop.so or hadoop.dll in one place in hadoop-project/pom.xml, to be inherited with
the correct value in each sub-module.  However, this doesn't seem feasible, because Maven
evaluates variables after evaluating inheritance.  For example, this means that even if hadoop-project/pom.xml
refers to basedir, then it evaluates to the basedir of the child project, and not hadoop-project's

Considering that, it seems we're forced into doing a bit of duplication.  Here is an idea
that will at least minimize the amount of duplication:

# hadoop-project/pom.xml defines a property, hadoop.common.build.dir, set to basedir/../../hadoop-common-project/hadoop-common/target.
# hadoop-project/pom.xml defines LD_LIBRARY_PATH and PATH in terms of hadoop.common.build.dir.
# For the majority of sub-modules, the default value will be correct, because most sub-modules
have a depth of 2 in the source tree.
# For sub-modules at a different depth, change its pom.xml to provide a different definition
for hadoop.common.build.dir.  I believe the overrides needed would be:
## hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml needs to use ../../.. in the relative
## hadoop-yarn-project/hadoop-yarn needs to use ../../.. in the relative path
## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications needs to use ../../../.. in the
relative path
## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server needs to use ../../../.. in the relative

If my understanding of pom.xml inheritance is correct (which is not guaranteed!), then evaluation
of LD_LIBRARY_PATH and PATH will end up using the child's definition of hadoop.common.build.dir.
 This way, at least we won't have to duplicate the entire maven-surefire-plugin configuration
in each sub-module.

> Some tests cannot load native library
> -------------------------------------
>                 Key: HADOOP-9290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, native
>    Affects Versions: 3.0.0, trunk-win
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HADOOP-9290.patch
> Some tests are unable to load the native DLL on Windows. The culprit appears to be an
incorrect PATH configuration for the native-win profile in hadoop-project/pom.xml.
> {code:xml}
>     <profile>
>       <id>native-win</id>
>       <activation>
>         <os>
>           <family>Windows</family>
>         </os>
>       </activation>
>       <build>
>         <plugins>
>           <plugin>
>             <groupId>org.apache.maven.plugins</groupId>
>             <artifactId>maven-surefire-plugin</artifactId>
>             <configuration>
>               <environmentVariables>
>                 <!-- Specify where to look for the native DLL on Windows -->
>                 <PATH>${env.PATH};${basedir}\..\..\hadoop-common-project\hadoop-common\target\bin;</PATH>
>               </environmentVariables>
>             </configuration>
>           </plugin>
>         </plugins>
>       </build>
>     </profile>
> {code}
> This is evaluated independently by each project and the relative path is not always correct.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message