Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 69A7818D4C for ; Thu, 14 Jan 2016 18:57:30 +0000 (UTC) Received: (qmail 13646 invoked by uid 500); 14 Jan 2016 18:57:26 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 13525 invoked by uid 500); 14 Jan 2016 18:57:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 13514 invoked by uid 99); 14 Jan 2016 18:57:26 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2016 18:57:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C3ECEC05A2 for ; Thu, 14 Jan 2016 18:57:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id kAoNE3iDxbSW for ; Thu, 14 Jan 2016 18:57:15 +0000 (UTC) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com [209.85.217.181]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id CA5F220525 for ; Thu, 14 Jan 2016 18:57:14 +0000 (UTC) Received: by mail-lb0-f181.google.com with SMTP id bc4so301933719lbc.2 for ; Thu, 14 Jan 2016 10:57:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PWkemnYu6zFSYV0cfFkFVtfn2Xc56N1TbN7rSLTCZqA=; b=IjNMC5604sbaFodR2IJxVtLeErSUpI8ke2NGv5WkUGzv8JetGnD0tv39FYckDHDlCJ ZDo0YIZ0vFpmy6GYcl+ZyXk8txAlCmgjXQIb1bgubm4Bhc/kBBsQX9b65dgIoBrM0zik 888FwGDG2xxe1WVENOcwc0b1hCx38XMGvNozOVgKLEhgqTp3iUmAH5+uFg94dSorEvzg 5lCWfnTAptUFSrqKiyf/GvUY4wUeWhJaTZgocf8LX3mRoq2gVLuw5RwVNY5XUK76Lyw9 arclMf7M4vFHlbI0wf1cAkW5atznbVj6aoQeUwyhHmAOxPchHGQKHjEFbL0JfJ8LeX+W JcIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=PWkemnYu6zFSYV0cfFkFVtfn2Xc56N1TbN7rSLTCZqA=; b=k53on2mRP8XO8s2CTIyYtb1A3pBd5bqeIcux+uxnBjv7pXwpcXuTsEThCZX4F++8oS IBZTSW3zSFxFtEVz/pYL/y9C+X4QEiZpTEIMYA9WG77JU9BCr6onTh/Gmu4H6HxdZP7P GZG6mZ8vhLGDrc/qXqjYcOSgVTcAdINh8P9Po608d7tRyJhFWvZ85YDKQwCtjOW+v798 JXS9/8VMm0nA6u+w4OADBePal2UEryJPEAOjKY6nVDyqXot6HUq5idnwzvqxvNlx+9ju Y6TOmvmnbM7w4liUdm6tLylfMxhHAW3FozWOQc2a5/NBEvCdiJnRQsUztng0M5xAkcCN Za9Q== X-Gm-Message-State: ALoCoQn5OvckJzQ11sAS9dWoK9JnZ/RIEFdhvEZFPWn5gn7iodA3aepuDhyQaLyrAzvexxD5xBTm8y1b855kG+BcstK9uswqCA== MIME-Version: 1.0 X-Received: by 10.112.201.3 with SMTP id jw3mr1581961lbc.27.1452797833243; Thu, 14 Jan 2016 10:57:13 -0800 (PST) Received: by 10.25.155.78 with HTTP; Thu, 14 Jan 2016 10:57:13 -0800 (PST) Received: by 10.25.155.78 with HTTP; Thu, 14 Jan 2016 10:57:13 -0800 (PST) In-Reply-To: References: Date: Thu, 14 Jan 2016 10:57:13 -0800 Message-ID: Subject: Re: fs.s3a.endpoint not working From: Alexander Pivovarov To: "Phillips, Caleb" Cc: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c372d4c7f2af05294fdab7 --001a11c372d4c7f2af05294fdab7 Content-Type: text/plain; charset=UTF-8 http://www.jets3t.org/toolkit/configuration.html On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" wrote: > Add jets3t.properties file with s3service.s3-endpoint= to > /etc/hadoop/conf folder > > The folder with the file should be in HADOOP_CLASSPATH > > JetS3t library which is used by hadoop is looking for this file. > On Dec 22, 2015 12:39 PM, "Phillips, Caleb" > wrote: > >> Hi All, >> >> New to this list. Looking for a bit of help: >> >> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object >> store. >> >> This issue was discussed, but left unresolved, in this thread: >> >> >> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E >> >> And here, on Cloudera's forums (the second post is mine): >> >> >> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180 >> >> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using >> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files. >> >> However, when I point the fs.s3a.endpoint configuration directive at my >> non-AWS S3-Compatable object storage, it appears to still point at (and >> authenticate against) AWS. >> >> I've checked and double-checked my credentials and configuration using >> both Python's boto library and the s3cmd tool, both of which connect to >> this non-AWS data store just fine. >> >> Any help would be much appreciated. Thanks! >> >> -- >> Caleb Phillips, Ph.D. >> Data Scientist | Computational Science Center >> >> National Renewable Energy Laboratory (NREL) >> 15013 Denver West Parkway | Golden, CO 80401 >> 303-275-4297 | caleb.phillips@nrel.gov >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org >> For additional commands, e-mail: user-help@hadoop.apache.org >> >> --001a11c372d4c7f2af05294fdab7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

http://www.jets3t.org/toolkit/configuration.html

On Jan 14, 2016 10:56 AM, "Alexander Pivova= rov" <apivovarov@gmail.com<= /a>> wrote:

Add jets3t.properties file with s3service.s3-endpoint=3D<endpo= int> to /etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this f= ile.

On Dec 22, 2015 12:39 PM, "Phillips, Caleb&= quot; <Cale= b.Phillips@nrel.gov> wrote:
Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) objec= t store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_m= box/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnT= vNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integrati= on/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hado= op, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non= -AWS S3-Compatable object storage, it appears to still point at (and authen= ticate against) AWS.

I've checked and double-checked my credentials and configuration using = both Python's boto library and the s3cmd tool, both of which connect to= this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-2= 75-4297 | = caleb.phillips@nrel.gov

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org

--001a11c372d4c7f2af05294fdab7--