hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
Date Tue, 06 Jun 2017 21:41:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039692#comment-16039692
] 

Allen Wittenauer edited comment on HADOOP-14498 at 6/6/17 9:40 PM:
-------------------------------------------------------------------

OK, using 'hadoop --debug classpath' we can see exactly what is happening with the specific
construction of "hadoop-azure,hadoop-aws,hadoop-azure-datalake"

{code}
$ bin/hadoop --debug classpath 2>&1 | grep PROFILES
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure
DEBUG: HADOOP_SHELL_PROFILES accepted hdfs
DEBUG: HADOOP_SHELL_PROFILES accepted mapred
DEBUG: HADOOP_SHELL_PROFILES accepted yarn
{code}

hadoop-azure is getting rejected by the shell profiles code because it is getting up in the
dedupe pattern match code.  Converting this to use the new array code added in HADOOP-13595
will probably fix this.  Flipping the order will also make it pass.



was (Author: aw):
OK, using 'hadoop --debug classpath' we can see exactly what is happening with the specific
construction of "hadoop-azure,hadoop-aws,hadoop-azure-datalake"

{code}
$ bin/hadoop --debug classpath 2>&1 | grep PROFILES
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure
DEBUG: HADOOP_SHELL_PROFILES accepted hdfs
DEBUG: HADOOP_SHELL_PROFILES accepted mapred
DEBUG: HADOOP_SHELL_PROFILES accepted yarn
{code}

hadoop-azure is getting rejected by the shell profiles code because it is getting up in the
dedupe pattern match code.  Converting this to use the new array code added will probably
fix this.  Flipping the order will also make it pass.


> HADOOP_OPTIONAL_TOOLS not parsed correctly
> ------------------------------------------
>
>                 Key: HADOOP-14498
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14498
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Mingliang Liu
>            Priority: Critical
>
> # This will make hadoop-azure not show up in the hadoop classpath, though both hadoop-aws
and hadoop-azure-datalake are in the classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we make some assumptions
that hadoop tool modules have a single "-" in names, and the _hadoop-azure-datalake_ overrides
the _hadoop-azure_. Or any other assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message