hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7563) hadoop-config.sh setup HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME incorrectly
Date Tue, 23 Aug 2011 16:47:29 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089572#comment-13089572
] 

Alejandro Abdelnur commented on HADOOP-7563:
--------------------------------------------

Eric, the *compiled* scope in Maven actually means that the JAR will be bundled with the application,
it is a misleading name but I guess it stuck for historical reasons.

The assemblies are doing a funny logic to exclude/include JARs in lib, the purpose of using
wildcard instead the concrete artifacts to exclude/include is that the same assembly is used
by common & hdfs (as per your feedback), and eventually it should be leveraged by mapreduce.

Currently the assembly does:


{code}
  <fileSets>
    ...
    <fileSet>
      <directory>${project.build.directory}</directory>
      <outputDirectory>/share/hadoop/${hadoop.component}</outputDirectory>
      <includes>
        <include>${project.artifactId}-${project.version}.jar</include>
        <include>${project.artifactId}-${project.version}-tests.jar</include>
        <include>${project.artifactId}-${project.version}-sources.jar</include>
        <include>${project.artifactId}-${project.version}-test-sources.jar</include>
      </includes>
    </fileSet>
    ...
  </fileSets>
  <dependencySets>
    <dependencySet>
      <outputDirectory>/share/hadoop/${hadoop.component}/lib</outputDirectory>
      <unpack>false</unpack>
      <scope>runtime</scope>
      <useProjectArtifact>false</useProjectArtifact>
      <excludes>
        <exclude>org.apache.ant:*:jar</exclude>
        <exclude>org.apache.hadoop:hadoop-*:jar</exclude>
        <exclude>jdiff:jdiff:jar</exclude>
      </excludes>
    </dependencySet>
  </dependencySets>
{code}

IMO, the assembly should do:

{code}
  <fileSets>
    ...
  </fileSets>
  <dependencySets>
    <dependencySet>
      <outputDirectory>/share/hadoop/${hadoop.component}/lib</outputDirectory>
      <unpack>false</unpack>
      <scope>runtime</scope>
      <useProjectArtifact>false</useProjectArtifact>
      <excludes>
        <exclude>org.apache.ant:*:jar</exclude>
        <exclude>jdiff:jdiff:jar</exclude>
      </excludes>
    </dependencySet>
  </dependencySets>
{code}

What's different?

* Hadoop JARs don't have an special handling, they end up in lib/
* Hadoop 'tests' JAR is not bundled with the distribution (*)
* Hadoop 'sources' JARs are not bundled with the distribution (*)

(*: they are all published to Apache Maven repository, and IDEs download them from there automatically,
the IDE will not look at the distribution location)


> hadoop-config.sh setup HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME incorrectly
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7563
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7563-1.patch, HADOOP-7563.patch
>
>
> HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME was set to HADOOP_PREFIX/share/hadoop/hdfs and
HADOOP_PREFIX/share/hadoop/mapreduce.  This setup confuses the location of hdfs and mapred
scripts.  Instead the script should look for hdfs and mapred script in HADOOP_PREFIX/bin.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message