hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Aquilina <jaquil...@eagleeyet.net>
Subject Re: fs.s3a.endpoint not working
Date Thu, 14 Jan 2016 12:48:03 GMT
Im not totally following this thread from the beginning but I might be
able to help as I have some experience with Amazon EMR (elastic map
reduce) when working with custom jar files and s3 

Are you using EMR or something internal and offloading strage to s3?

Jonathan Aquilina

On 2016-01-13 23:21, Phillips, Caleb wrote:

> Hi Billy (and others),
> One of the threads suggested using the core-site.xml. Did you try putting your configuration
in there?
> Yes, I did try that. I've also tried setting it dynamically in e.g., spark. I can verify
that it is getting the configuration correctly:
> hadoop org.apache.hadoop.conf.Configuration
> Still it never connects to our internal S3-compatable store and always connects to AWS.
> One thing I've noticed is that the AWS stuff is handled by an underlying library (I think
jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff
and spelunking through the hadoop code, I kept running into blocks with that library.
> I started digging into the code. I found that the custom endpoint was introduced with
this patch:
> https://issues.apache.org/jira/browse/HADOOP-11261
> It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem
> That code calls this function in the AWS Java SDK:
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
> However, no matter what configuration I use, it still seems to connect to Amazon AWS.
Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so,
it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.
> If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable
store, please chime in!
> William Watson
> Software Engineer
> (904) 705-7056 PCS
> On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Caleb.Phillips@nrel.gov<mailto:Caleb.Phillips@nrel.gov>>
> Hi All,
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it'd be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<mailto:caleb.phillips@nrel.gov>
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Caleb.Phillips@nrel.gov<mailto:Caleb.Phillips@nrel.gov>>
>> Hi All,
>> New to this list. Looking for a bit of help:
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>> This issue was discussed, but left unresolved, in this thread:
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>> Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<mailto:Au5Es_fLUgZMGwkkgA3JyA1ASi3u%2BisJCuYmfnTvNkGuQ@mail.gmail.com>%3E
>> And here, on Cloudera's forums (the second post is mine):
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>> nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>> Any help would be much appreciated. Thanks!
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov<mailto:caleb.phillips@nrel.gov>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<mailto:user-unsubscribe@hadoop.apache.org>
>> For additional commands, e-mail: user-help@hadoop.apache.org<mailto:user-help@hadoop.apache.org>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
View raw message