Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: hadoop 3 scripts & classpath setup
From: Allen Wittenauer <aw@effectivemachines.com>
In-Reply-To: <A231770D-4730-4CFC-9267-2261861897BA@hortonworks.com>
Date: Tue, 22 Aug 2017 09:24:56 -0700
Cc: Hadoop Common <common-dev@hadoop.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <E55E9810-044D-4C3A-BECC-3692BB07C63F@effectivemachines.com>
References: <A231770D-4730-4CFC-9267-2261861897BA@hortonworks.com>
To: Steve Loughran <stevel@hortonworks.com>
archived-at: Tue, 22 Aug 2017 16:25:11 -0000


> On Aug 22, 2017, at 6:00 AM, Steve Loughran <stevel@hortonworks.com> =
wrote:
>=20
>=20
> I'm having problems getting the s3 classpath setup on the CLI & am =
trying to work out what I'm doing wrong.
>=20
>=20
> without setting things up, you can't expect to talk to blobstores
>=20
> hadoop fs -ls wasb://something/
> hadoop fs -ls s3a://landsat-pds/
>=20
> That's expected.

	Yup.

> but what I can't do is get the aws bits on the CP via =
HADOOP_OPTIONAL_TOOLS
>=20
> export =
HADOOP_OPTIONAL_TOOLS=3D"hadoop-azure,hadoop-aws,hadoop-adl,hadoop-opensta=
ck"
>=20
> Once I do that the wasb:// ls works (or at least doesnt throw a CNFE), =
but the s3a URL still fails

	Hmm. So HOT is getting processed at least somewhat then...

> if Add the line to ~/.hadooprc all becomes well
>=20
> hadoop_add_to_classpath_tools hadoop-aws
>=20
> any ideas?

	Setting HOT should be calling the equivalent of =
hadoop_add_to_classpath_tools hadoop-aws in the code path.  Luckily, we =
have debugging tools in 3.x[1]:

First, let=92s duplicate the failure conditions, but only activate =
hadoop-aws since it should be standalone and cuts our output down:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
$ cat ~/.hadooprc
cat: /Users/aw/.hadooprc: No such file or directory
$ bin/hadoop envvars | grep CONF
HADOOP_CONF_DIR=3D'/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/etc/hadoop'
$ pwd
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT
$ grep OPTIONAL_TOOLS etc/hadoop/hadoop-env.sh
# export =
HADOOP_OPTIONAL_TOOLS=3D"hadoop-aliyun,hadoop-aws,hadoop-azure,hadoop-azur=
e-datalake,hadoop-kafka,hadoop-openstack"
export HADOOP_OPTIONAL_TOOLS=3D"hadoop-aws=94
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Using --debug, let=92s see what happens:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
$ bin/hadoop --debug classpath 2>&1 | egrep '(tools|hadoop-aws)'
DEBUG: shellprofiles: =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-aliyun.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-archive-logs.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-archives.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-aws.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-azure-datalake.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-azure.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-distcp.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-extras.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-gridmix.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-hdfs.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-httpfs.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-kafka.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-kms.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-mapreduce.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-openstack.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-rumen.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-streaming.sh =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-yarn.sh
DEBUG: Adding hadoop-aws to HADOOP_TOOLS_OPTIONS
DEBUG: Profiles: importing =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/bin/../libexec/shellprofile.d/hado=
op-aws.sh
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: Profiles: hadoop-aws classpath
DEBUG: Append CLASSPATH: =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/aws-java-sd=
k-bundle-1.11.134.jar
DEBUG: Append CLASSPATH: =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/java-xmlbui=
lder-0.4.jar
DEBUG: Append CLASSPATH: =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/jets3t-0.9.=
0.jar
DEBUG: Append CLASSPATH: =
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-=
3.0.0-beta1-SNAPSHOT.jar
=
/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/etc/hadoop:/Users/aw/H/hadoop-3.0.=
0-beta1-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/H/hadoop-3.0.0-beta1-=
SNAPSHOT/share/hadoop/common/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/sha=
re/hadoop/tools/lib/aws-java-sdk-bundle-1.11.134.jar:/Users/aw/H/hadoop-3.=
0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/java-xmlbuilder-0.4.jar:/Users/a=
w/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/jets3t-0.9.0.jar:/U=
sers/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.=
0.0-beta1-SNAPSHOT.jar:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoo=
p/hdfs:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/hdfs/lib/*:/Us=
ers/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/H/hadoo=
p-3.0.0-beta1-SNAPSHOT/share/hadoop/mapreduce/*:/Users/aw/H/hadoop-3.0.0-b=
eta1-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPS=
HOT/share/hadoop/yarn/*
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

OK, the =93extra=94 bits are definitely getting added.  With the =
addition of the debug lines:
* the hadoop-aws profile and tools hooks are getting executed
* the hadoop-aws classpath function is getting executed (aka =
hadoop_add_to_classpath_tools hadoop-aws)
* the classpath isn=92t rejecting any jars
* the final line definitely has AWS there.

So we should be good to go assuming the profile and supplemental tools =
code is correct.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
$ bin/hadoop fs -ls s3a://landsat-pds/
ls: Interrupted
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

umm, ok?  No CNFE though.  If I disable the network:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
$ bin/hadoop fs -ls s3a://landsat-pds/
ls: doesBucketExist on landsat-pds: com.amazonaws.AmazonClientException: =
No AWS Credentials provided by BasicAWSCredentialsProvider =
EnvironmentVariableCredentialsProvider =
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: =
Unable to load credentials from service endpoint: No AWS Credentials =
provided by BasicAWSCredentialsProvider =
EnvironmentVariableCredentialsProvider =
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: =
Unable to load credentials from service endpoint
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Ugly error, but still no CNFE. So at least out of the box with a build =
from last week. I guess this is working?  At this point, it=92d probably =
be worthwhile to make sure that the libexec/shellprofile.d/hadoop-aws.sh =
on your system is in working order. In particular...

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
if hadoop_verify_entry HADOOP_TOOLS_OPTIONS "hadoop-aws"; then
  hadoop_add_profile "hadoop-aws=94
fi
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=85 is the magic code.  It (effectively[2]) says that if =
HADOOP_OPTIONAL_TOOLS has hadoop-aws in it, then activate the hadoop-aws =
profile which should end up calling hadoop_add_to_classpath_tools =
hadoop-aws.   Might also be worthwhile to check simple stuff like =
permissions.

[1] It=92s tempting to say =93now=94, but given that debug was added =
several years ago. it=92s more like branch-2 is just really ancient =
rather than 3.x being "current".

[2]  yes, that variable is supposed to be HADOOP_TOOLS_OPTIONS.  HOT =
gets transformed into HADOOP_OPTIONAL_TOOLS  internally for =93reasons=94.=
  It=92s a longer discussion that most people aren=92t interested in.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org