flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiwan Park <chiwanp...@apache.org>
Subject Re: S3 as streaming source
Date Fri, 03 Jun 2016 06:23:17 GMT
Hi all,

I think we can use `readFile`, `readFileStream` methods in `StreamExecutionEnvironment` to
create streaming source from S3 because data are stored as file in S3. But I haven’t test
it.

Regards,
Chiwan Park

> On Jun 3, 2016, at 2:37 PM, Tzu-Li (Gordon) Tai <tzulitai@gmail.com> wrote:
> 
> Hi Soumya,
> 
> No, currently there is no Flink standard supported S3 streaming source. As
> far as I know, there isn't one out in the public yet either. The community
> is open to submissions for new connectors, so if you happen to be working on
> one for S3, you can file up a JIRA to let us know.
> 
> Also, are you looking for a S3 streaming source that fetches S3 event
> notifications (ref:
> http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html), or
> streaming files / objects from S3 for a data stream program? I assume the
> first one, since otherwise writing Flink batch jobs will suit you more (the
> batch DataSet API already supports this).
> 
> 
> 
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/S3-as-streaming-source-tp7357p7358.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.


Mime
View raw message