flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Morin <st...@stevemorin.com>
Subject Flink work with raw S3 (S3FileSystem or other), not a HDFS backed by S3 (S3AFileSystem, NativeS3FileSystem)?
Date Fri, 11 Nov 2016 16:35:16 GMT
Use-case: I am trying to see how to use flink with s3, where we use our own
client libraries or things like AWS firehose to put data into S3, then
process it in batch using flink.  This clients are putting data into S3
with out HDFS - Aka we aren't using HDFS on top of S3.

Most of what I can find referenced [1] is using HDFS backed by S3 (
S3AFileSystem, NativeS3FileSystem)

I find one reference [2] that using S3 Filesystem (S3FileSystem) doesn't wo
rk.

Can anyone with Flink experience help give any insight on this?

References:

   - [1] -
   https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/aws.html
   - [2] -
   http://stackoverflow.com/questions/32959790/run-apache-flink-with-amazon-s3


-- 
*Steve Morin | Managing Partner - CTO*

*Nvent*

O 800-407-1156 ext 803 <800-407-1156;803> | M 347-453-5579

smorin@nventdata.com  <smorin@nventdata.com>

*Enabling the Data Driven Enterprise*
*(Ask us how we can setup scalable open source realtime billion+ event/data
collection/analytics infrastructure in weeks)*

Service Areas: Management & Strategy Consulting | Data Engineering | Data
Science & Visualization

Mime
View raw message