flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Processing S3 data with Apache Flink
Date Tue, 06 Oct 2015 02:13:41 GMT
Hi Kostia,

thank you for writing to the Flink mailing list. I actually started to try
out our S3 File system support after I saw your question on StackOverflow
I found that our S3 connector is very broken. I had to resolve two more
issues with it, before I was able to get the same exception you reported.

Another Flink commiter looked into the issue as well (it was confirmed as
well) but there was no solution [2].

So for now, I would say we have to assume that our S3 connector is not
working. I will start a separate discussion at the developer mailing list
to remove our S3 connector.

The good news is that you can just use Hadoop's S3 File System
implementation with Flink.

I used this Flink program to verify its working:

public class S3FileSystem {
   public static void main(String[] args) throws Exception {
      ExecutionEnvironment ee = ExecutionEnvironment.createLocalEnvironment();
      DataSet<String> myLines =

also, you need to make a Hadoop configuration file available to Flink.
When running flink locally in your IDE, just create a "core-site.xml" in
the src/main/resource folder, with the following content:




Maybe you are running on a cluster, then re-use the existing core-site.xml
file (= edit it) and point to the directory using Flink's
fs.hdfs.hadoopconf configuration option.

With these two things in place, you should be good to go.


On Mon, Oct 5, 2015 at 8:19 PM, Kostiantyn Kudriavtsev <
kudryavtsev.konstantin@gmail.com> wrote:

> Hi guys,
> I,m trying to get work Apache Flink 0.9.1 on EMR, basically to read
> data from S3. I tried the following path for data
> s3://mybucket.s3.amazonaws.com/folder, but it throws me the following
> exception:
> java.io.IOException: Cannot establish connection to Amazon S3:
> com.amazonaws.services.s3.model.AmazonS3Exception: The request signature
> we calculated does not match the signature you provided. Check your key
> and signing method. (Service: Amazon S3; Status Code: 403;
> I added access and secret keys, so the problem is not here. I=92m using
> standard region and gave read credential to everyone.
> Any ideas how can it be fixed?
> Thank you in advance,
> Kostia

View raw message