hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-14142) S3A - Adding unexpected prefix
Date Fri, 03 Mar 2017 12:00:49 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran resolved HADOOP-14142.
    Resolution: Invalid

And closing as invalid. It's your home directory, which is the default unless you put a trailing
"/" on the path. 

As stated before, If you are trying to show your S3 implementation works with hadoop, run
the hadoop-aws test suite first. For spark, [try these tests|https://github.com/steveloughran/spark-cloud-examples]

Either way, immediately panicing with critical bug reports isn't going to fix the problem.
You've got partway there with the logging, but take advantage of the code: step through it
to see what's happening. It's what we end up doing

> S3A - Adding unexpected prefix
> ------------------------------
>                 Key: HADOOP-14142
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14142
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.3
>            Reporter: Vishnu Vardhan
>            Priority: Minor
> Hi:
> S3A seems to prefix unexpected prefix to my s3 path
> Specifically, in the debug log below the following line is unexpected
> >  GET /myBkt8/?max-keys=1&prefix=user%2Fvardhan%2F&delimiter=%2F HTTP/1.1
> It is not clear where the "prefix" is coming from and why.
> I executed the following commands
> sc.setLogLevel("DEBUG")
> sc.hadoopConfiguration.set("fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem")
> sc.hadoopConfiguration.set("fs.s3a.endpoint","webscaledemo.netapp.com:8082")
> sc.hadoopConfiguration.set("fs.s3a.access.key","")
> sc.hadoopConfiguration.set("fs.s3a.secret.key","")
> sc.hadoopConfiguration.set("fs.s3a.path.style.access","false")
> val s3Rdd = sc.textFile("s3a://myBkt98")
> s3Rdd.count()
> ----

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message