flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Problem with Amazon S3
Date Wed, 03 Jun 2015 17:17:38 GMT
Flink allows to use Hadoop's FileSystem interface as well [1].

Hadoop actually ships a s3 file system implementation by default, and I
suspect its in a better shape than Flink's implementation. Maybe it would
make sense to use Hadoop's S3 implementation through Flink's Hadoop FS
support.

Please let me know if you are facing any issues while using this approach.

[1]
http://ci.apache.org/projects/flink/flink-docs-master/apis/example_connectors.html

On Fri, May 22, 2015 at 6:27 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> I am getting the same error as you are. Investigating now.
>
> On Wed, May 20, 2015 at 12:34 PM, pietro <pietro.pinoli@gmail.com> wrote:
> > Dears,
> >
> > I am still having problem retriving data from the S3. I followed all you
> > indication in the previous posts, but now I get this error:
> >
> > 15/05/20 10:47:05 INFO s3.S3FileSystem: Creating new S3 file system
> binding
> > with Reduced Redundancy Storage enabled
> > 15/05/20 10:47:13 WARN io.DelimitedInputFormat: Could not determine
> > statistics for file 's3://
> genomic.s3-us-west-2.amazonaws.com/flink/ref/meta'
> > due to an io error: Cannot establish connection to Amazon S3:
> > com.amazonaws.services.s3.model.AmazonS3Exception: The request signature
> we
> > calculated does not match the signature you provided. Check your key and
> > signing method. (Service: Amazon S3; Status Code: 403; Error Code:
> > SignatureDoesNotMatch; Request ID: E719C84656C22D70), S3 Extended Request
> > ID:
> >
> 5yE3QhMxlrVuCiPe5lN/cVAWptceXRNuSUmIG9kwRtioimOX3znU4Fj3aY7+P1MTR4BTecyTvVM=
> >
> >
> >
> > I checked the keys in the flink-conf.yaml and they are correct.
> >
> > Any idea?
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Problem-with-Amazon-S3-tp946p1310.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>

Mime
View raw message