hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-15248) 400 Bad Request while trying to access S3 through Spark
Date Wed, 21 Feb 2018 10:55:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran resolved HADOOP-15248.
-------------------------------------
    Resolution: Invalid

I'm going to close this as "Invalid" as JIRA isn't a place to deal with support requests.
Sorry. You are going to have to fend for yourself. Either It's a configuration problem or
you've got an incompatbile  joda-time/JVM combo.

* This is an old version of Hadoop; we've done a lot with functionality and performance in
S3, particularly in Hadoop 2.8. Upgrade Hadoop and try again.
* There is an [S3A Troubleshooting doc|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md]
* And: in [cloudstore|https://github.com/steveloughran/cloudstore], a tool for looking at
your settings and trying to bootstrap connectivity.

LIke I said, you are on your own. These are exactly the docs and tools we use to identify
and fix our own issues.

> 400 Bad Request while trying to access S3 through Spark
> -------------------------------------------------------
>
>                 Key: HADOOP-15248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15248
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.3
>         Environment: macOS 10.13.3 (17D47)
> Spark 2.2.1
> Hadoop 2.7.3
>            Reporter: Farshid
>            Priority: Minor
>
>  
> I'm trying to read a file thorugh {{s3a}} from a bucket in us-east-2 (Ohio) and I'm
getting 400 Bad Request response:
> _com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon
S3, AWS Request ID: [removed], AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended
Request ID: [removed]_
> Since my code works with another bucket in Sydney, it seems to be a signing API version
issue (Ohio supports only 4, Sydney supports 2 and 4). So I tried setting the endpoint by
adding this to {{spark-submit}} as suggested in other posts:
> _--conf "spark.hadoop.fs.s3a.endpoint=s3.us-east-2.amazonaws.com"_ 
> But that didn't make any difference. I also tried adding the same to a conf file and
passing it using {{--properties-file [file_path]}}
> _spark.hadoop.fs.s3a.endpoint               s3.us-east-2.amazonaws.com_
> No difference. I still get the same error for Ohio (and it doesn't work with Sydney any
more, for obvious reasons).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message