flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Morin <steve.mo...@gmail.com>
Subject Re: Reading files from an S3 folder
Date Wed, 23 Nov 2016 18:34:34 GMT
Alex,
  We are working on the same thing, for the same exact reason.  We are
trying to avoid the complexities of running HDFS just for the file
storage.  We are also okay with the S3 limitations it introduces.  We'll
try and update the group if we find solutions for parallelizing the files
consumption.

Scott,
  What problems did you encounter trying to save flink state and savepoint
data?

Flink checkpoint & savepoint data. Like you, I tried using S3 to persist
> Flink state, but encountered AWS SDK issues and felt like I was going down
> an ill-advised path


-Steve


-- 
Steve Morin | Hacker, Entrepreneur, Startup Advisor
twitter.com/SteveMorin | stevemorin.com
*Live the dream start a startup. Make the world ... a better place.*

Mime
View raw message