flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Flink work with raw S3 (S3FileSystem or other), not a HDFS backed by S3 (S3AFileSystem, NativeS3FileSystem)?
Date Mon, 14 Nov 2016 09:14:41 GMT
The Flink docs show how to setup Flink's internal file system operations to use the S3FileSystem
(the StackOverflow question actually shows that it is working, see answer there).\

This is configuration is independent of what you are doing in your user code. If you want
to use your own S3 based clients, you will have to package them with your user code JARs.
Then you should be ready to go. That how you want to use them, right?

– Ufuk

On 11 November 2016 at 17:35:22, Steve Morin (steve@stevemorin.com) wrote:
> Use-case: I am trying to see how to use flink with s3, where we use our own
> client libraries or things like AWS firehose to put data into S3, then
> process it in batch using flink. This clients are putting data into S3
> with out HDFS - Aka we aren't using HDFS on top of S3.
>  
> Most of what I can find referenced [1] is using HDFS backed by S3 (
> S3AFileSystem, NativeS3FileSystem)
>  
> I find one reference [2] that using S3 Filesystem (S3FileSystem) doesn't wo
> rk.
>  
> Can anyone with Flink experience help give any insight on this?
>  
> References:
>  
> - [1] -
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/aws.html  
> - [2] -
> http://stackoverflow.com/questions/32959790/run-apache-flink-with-amazon-s3  
>  
>  
> --
> *Steve Morin | Managing Partner - CTO*
>  
> *Nvent*
>  
> O 800-407-1156 ext 803 <800-407-1156;803> | M 347-453-5579
>  
> smorin@nventdata.com  
>  
> *Enabling the Data Driven Enterprise*
> *(Ask us how we can setup scalable open source realtime billion+ event/data
> collection/analytics infrastructure in weeks)*
>  
> Service Areas: Management & Strategy Consulting | Data Engineering | Data
> Science & Visualization
>  


Mime
View raw message