Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AE93418187 for ; Wed, 17 Feb 2016 18:12:09 +0000 (UTC) Received: (qmail 10971 invoked by uid 500); 17 Feb 2016 18:12:05 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 10858 invoked by uid 500); 17 Feb 2016 18:12:05 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 10846 invoked by uid 99); 17 Feb 2016 18:12:05 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Feb 2016 18:12:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 897BCC08BB for ; Wed, 17 Feb 2016 18:12:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 5wAR-JaS6lHC for ; Wed, 17 Feb 2016 18:12:02 +0000 (UTC) Received: from mail-ob0-f173.google.com (mail-ob0-f173.google.com [209.85.214.173]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 50E1066A8D for ; Wed, 17 Feb 2016 14:35:29 +0000 (UTC) Received: by mail-ob0-f173.google.com with SMTP id jq7so16389420obb.0 for ; Wed, 17 Feb 2016 06:35:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=NyOXxdC2A1Syl+P5xJ3RUZwaN5DRHC/SX8NpX474yyw=; b=pDavQ+X12jmERyggtTQPfzY0ci7KZmXhBXCZ5iddLNbcu2avLc+tTLRs7K7Swfxgd9 +xXCInB8pxY0VafXG6cIeQqcs7VrbGwUW8VlN7+TJgvO5gK65YqCgXiMehKnmxIFk65C 4o0EBMKnPsTU5rcT+6YyvkwFxhPvpR6gQSx8Y0JRg8nJHq2RmBgCiE/09+WdJePvtQ6+ Cx6rAAO2aZLkjwMScAIOm2J9PLiyU0kScee7CQiIYyK5uPyfnegYLlmR5gX1hWCxmGhh nAMuhBXnU/s0PCe7ZnrIQKD3BJX4s7WOsiSu7GMttHglpsnL5lqd/4CBrR+XifIWl0a+ UVPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=NyOXxdC2A1Syl+P5xJ3RUZwaN5DRHC/SX8NpX474yyw=; b=EbWOv6rf7XgZJPU+W0XohpCLb9oNg8m+pWtFWoGwqqCkpZwvkgvuouFqX7OrOjKuJR DEB7B3Vj3Q6jorMDq43cIlLPqtrWCX3W48OGFDk9LxHlx0acY2A2eMmP40nzZXraMXdZ 3yQmiRhUlYPjcA2dUTrdGW4Rv66aQARnDDcxECxGTG20c2ntGLiDaTWXT4QyyDeo7GeD /0dAWZPVdiubEBsTrzgwX5c8OVWZFw/XQdZ/Ag5skd/Ytdkxl/EIxcajDPkiL/A957Y0 /LwrttGUM6LQHgXU9AlGmomEpd5J0B9ZEyHGjTQ3WvTJcloiw/9BMKkr/vmt1ulMzBdw JfXg== X-Gm-Message-State: AG10YOTat78wKaS+xnRBZMHtE1wZqqg/O6PA5mlfqPoPhGG/HmLpQ4pLTMXRrciwnEQ3UUuePi58EB5hOW5NkQ== X-Received: by 10.60.38.169 with SMTP id h9mr1489320oek.52.1455719728204; Wed, 17 Feb 2016 06:35:28 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.169.72 with HTTP; Wed, 17 Feb 2016 06:35:08 -0800 (PST) In-Reply-To: References: From: Billy Watson Date: Wed, 17 Feb 2016 09:35:08 -0500 Message-ID: Subject: Re: fs.s3a.endpoint not working To: "Phillips, Caleb" Cc: Alexander Pivovarov , "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=089e0149530a4ad3fd052bf8291c --089e0149530a4ad3fd052bf8291c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks for following up! William Watson Lead Software Engineer On Tue, Feb 16, 2016 at 5:08 PM, Phillips, Caleb wrote: > Hi All, > > Just wanted to follow on that we got this working with the help of the > object storage vendor. After running in circles for a bit, the issue seem= s > to have been as simple as using the correct FQDN in the endpoint fields a= nd > disabling SSL. We used the jet3st properties, but it turns out those aren= =E2=80=99t > actually needed with recent Hadoop versions (?). > > For anyone who might be having similar issues, here are the relevant > configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1: > > > > > > fs.s3n.awsAccessKeyId > AWS access key ID > yourusername > > > > fs.s3n.awsSecretAccessKey > AWS secret key > sweetpassword > > > > fs.s3n.endpoint > youre.fqdn.here > > > > fs.s3n.ssl.enabled > false > > > > > > fs.s3a.access.key > AWS access key ID. Omit for Role-based > authentication. > yourusername > > > > fs.s3a.secret.key > AWS secret key. Omit for Role-based > authentication. > sweetpassword > > > > fs.s3a.connection.ssl.enabled > false > Enables or disables SSL connections to S3. > > > > fs.s3a.endpoint > AWS S3 endpoint to connect to. An up-to-date list is > provided in the AWS Documentation: regions and endpoints. Without thi= s > property, the standard region (s3.amazonaws.com) is assumed. > > your.fqdn.here > > > > > Also, as mentioned previously in the thread, it=E2=80=99s necessary to ad= d some > things to your HADOOP_CLASSPATH: > > export > HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/t= ools/lib/* > > You can test by: > > s3cmd mb s3://some-bucket # <- note that you have to do thi= s > with s3cmd, not hadoop, at least with our object store > hadoop fs -ls s3n://some-bucket/ > hadoop fs -ls s3a://some-bucket/ > hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/ > > HTH, > > -- > Caleb Phillips, Ph.D. > Data Scientist | Computational Science Center > > National Renewable Energy Laboratory (NREL) > 15013 Denver West Parkway | Golden, CO 80401 > 303-275-4297 | caleb.phillips@nrel.gov > > From: Billy Watson williamrwatson@gmail.com>> > Date: Tuesday, January 19, 2016 at 8:41 AM > To: Alexander Pivovarov >> > Cc: Caleb Phillips >, > "user@hadoop.apache.org" < > user@hadoop.apache.org> > Subject: Re: fs.s3a.endpoint not working > > Stupid question, I assume you're using a URL that starts with s3a and tha= t > your custom endpoint supports s3a? > > William Watson > Lead Software Engineer > > On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov > wrote: > > http://www.jets3t.org/toolkit/configuration.html > > On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" > wrote: > > Add jets3t.properties file with s3service.s3-endpoint=3D to > /etc/hadoop/conf folder > > The folder with the file should be in HADOOP_CLASSPATH > > JetS3t library which is used by hadoop is looking for this file. > > On Dec 22, 2015 12:39 PM, "Phillips, Caleb" > wrote: > Hi All, > > New to this list. Looking for a bit of help: > > I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object > store. > > This issue was discussed, but left unresolved, in this thread: > > > https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W= _Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E > > And here, on Cloudera's forums (the second post is mine): > > > https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpo= int-ignored-in-hdfs-site-xml/m-p/33694#M1180 > > I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop= , > I'm able to connect to S3 on AWS, and e.g., list/put/get files. > > However, when I point the fs.s3a.endpoint configuration directive at my > non-AWS S3-Compatable object storage, it appears to still point at (and > authenticate against) AWS. > > I've checked and double-checked my credentials and configuration using > both Python's boto library and the s3cmd tool, both of which connect to > this non-AWS data store just fine. > > Any help would be much appreciated. Thanks! > > -- > Caleb Phillips, Ph.D. > Data Scientist | Computational Science Center > > National Renewable Energy Laboratory (NREL) > 15013 Denver West Parkway | Golden, CO 80401 > 303-275-4297 | caleb.phillips@nrel.gov caleb.phillips@nrel.gov> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org user-unsubscribe@hadoop.apache.org> > For additional commands, e-mail: user-help@hadoop.apache.org user-help@hadoop.apache.org> > > > --089e0149530a4ad3fd052bf8291c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for following up!
<= br clear=3D"all">
=
William Watson
Lead Software Engi= neer

On Tue, Feb 16, 2016 at 5:08 PM, Phillips, C= aleb <Caleb.Phillips@nrel.gov> wrote:
Hi All,

Just wanted to follow on that we got this working with the help of the obje= ct storage vendor. After running in circles for a bit, the issue seems to h= ave been as simple as using the correct FQDN in the endpoint fields and dis= abling SSL. We used the jet3st properties, but it turns out those aren=E2= =80=99t actually needed with recent Hadoop versions (?).

For anyone who might be having similar issues, here are the relevant config= uration in core-site.xml for S3A and S3N with Hadoop 2.7.1:

<configuration>

<!-- S3N Connector to Obsidian -->
<property>
=C2=A0<name>fs.s3n.awsAccessKeyId</name>
=C2=A0<description>AWS access key ID</description>
=C2=A0<value>yourusername</value>
</property>

<property>
=C2=A0<name>fs.s3n.awsSecretAccessKey</name>
=C2=A0<description>AWS secret key</description>
=C2=A0<value>sweetpassword</value>
</property>

<property>
=C2=A0<name>fs.s3n.endpoint</name>
=C2=A0<value>youre.fqdn.here</value>
</property>

<property>
=C2=A0<name>fs.s3n.ssl.enabled</name>
=C2=A0<value>false</value>
</property>

<!-- S3A Connector to Obsidian -->

<property>
=C2=A0<name>fs.s3a.access.key</name>
=C2=A0<description>AWS access key ID. Omit for Role-based authenticat= ion.</description>
=C2=A0<value>yourusername</value>
</property>

<property>
=C2=A0<name>fs.s3a.secret.key</name>
=C2=A0<description>AWS secret key. Omit for Role-based authentication= .</description>
=C2=A0<value>sweetpassword</value>
</property>

<property>
=C2=A0<name>fs.s3a.connection.ssl.enabled</name>
=C2=A0<value>false</value>
=C2=A0<description>Enables or disables SSL connections to S3.</des= cription>
</property>

<property>
=C2=A0<name>fs.s3a.endpoint</name>
=C2=A0<description>AWS S3 endpoint to connect to. An up-to-date list = is
=C2=A0 =C2=A0 provided in the AWS Documentation: regions and endpoints. Wit= hout this
=C2=A0 =C2=A0 property, the standard region (s3.amazonaws.com) is assumed= .
=C2=A0</description>
=C2=A0<value>your.fqdn.here</value>
</property>

</configuration>

Also, as mentioned previously in the thread, it=E2=80=99s necessary to add = some things to your HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/had= oop/tools/lib/*

You can test by:

s3cmd mb s3://some-bucket=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0# <- note that you have to do this with s3cmd, not hadoop, at leas= t with our object store
hadoop fs -ls s3n://some-bucket/
hadoop fs -ls s3a://some-bucket/
hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/

HTH,

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297=C2=A0|= caleb.phillips@nrel.gov

From: Billy Watson <w= illiamrwatson@gmail.com<mailto:williamrwatson@gmail.com>>
Date: Tuesday, January 19, 2016 at 8:41 AM
To: Alexander Pivovarov <apivova= rov@gmail.com<mailto:apivova= rov@gmail.com>>
Cc: Caleb Phillips <caleb.phi= llips@nrel.gov<mailto:cal= eb.phillips@nrel.gov>>, "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: fs.s3a.endpoint not working

Stupid question, I assume you're using a URL that starts with s3a and t= hat your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovaro= v <apivovarov@gmail.com<m= ailto:apivovarov@gmail.com>&= gt; wrote:

http://www.jets3t.org/toolkit/configuration.html=

On Jan 14, 2016 10:56 AM, "Alexander Pivovarov= " <apivovarov@gmail.com= <mailto:apivovarov@gmail.com= >> wrote:

Add jets3t.properties file with s3service.s3-endpoint=3D<endpoint> to= /etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.

On Dec 22, 2015 12:39 PM, "Phillips, Caleb&quo= t; <Caleb.Phillips@nrel.gov<mailto:Caleb.Phillips@nrel= .gov>> wrote:
Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) objec= t store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_m= box/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnT= vNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integrati= on/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hado= op, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non= -AWS S3-Compatable object storage, it appears to still point at (and authen= ticate against) AWS.

I've checked and double-checked my credentials and configuration using = both Python's boto library and the s3cmd tool, both of which connect to= this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297= <tel:303-275-4297> | caleb.phillips@nrel.gov<= /a><mailto:caleb.phillips@nre= l.gov>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<mailto:user-unsubscribe@hadoop.apache.org> For additional commands, e-mail: user-help@hadoop.apache.org<mailto:user-help@hadoop.apache.org>



--089e0149530a4ad3fd052bf8291c--