Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 6 Jun 2017 21:41:18 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13077788.1496781014000.11148.1496785278190@Atlassian.JIRA>
In-Reply-To: <JIRA.13077788.1496781014000@Atlassian.JIRA>
References: <JIRA.13077788.1496781014000@Atlassian.JIRA> <JIRA.13077788.1496781014685@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not
 parsed correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 06 Jun 2017 21:41:24 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039692#comment-16039692 ] 

Allen Wittenauer edited comment on HADOOP-14498 at 6/6/17 9:40 PM:
-------------------------------------------------------------------

OK, using 'hadoop --debug classpath' we can see exactly what is happening with the specific construction of "hadoop-azure,hadoop-aws,hadoop-azure-datalake"

{code}
$ bin/hadoop --debug classpath 2>&1 | grep PROFILES
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure
DEBUG: HADOOP_SHELL_PROFILES accepted hdfs
DEBUG: HADOOP_SHELL_PROFILES accepted mapred
DEBUG: HADOOP_SHELL_PROFILES accepted yarn
{code}

hadoop-azure is getting rejected by the shell profiles code because it is getting up in the dedupe pattern match code.  Converting this to use the new array code added in HADOOP-13595 will probably fix this.  Flipping the order will also make it pass.


was (Author: aw):
OK, using 'hadoop --debug classpath' we can see exactly what is happening with the specific construction of "hadoop-azure,hadoop-aws,hadoop-azure-datalake"

{code}
$ bin/hadoop --debug classpath 2>&1 | grep PROFILES
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure
DEBUG: HADOOP_SHELL_PROFILES accepted hdfs
DEBUG: HADOOP_SHELL_PROFILES accepted mapred
DEBUG: HADOOP_SHELL_PROFILES accepted yarn
{code}

hadoop-azure is getting rejected by the shell profiles code because it is getting up in the dedupe pattern match code.  Converting this to use the new array code added will probably fix this.  Flipping the order will also make it pass.


> HADOOP_OPTIONAL_TOOLS not parsed correctly
> ------------------------------------------
>
>                 Key: HADOOP-14498
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14498
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Mingliang Liu
>            Priority: Critical
>
> # This will make hadoop-azure not show up in the hadoop classpath, though both hadoop-aws and hadoop-azure-datalake are in the classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we make some assumptions that hadoop tool modules have a single "-" in names, and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org