flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: Tweets Stream input for Flink
Date Tue, 28 Oct 2014 21:46:59 GMT
Dear Mustafa,

Flink has built-in support for stream processing providing connectors for a
number of input sources, the Twitter Streaming API amongst others. We
provide an example of reading from the publicly available tweets and
parsing the flat JSON objects to build our internal DataStream
representation containing the relevant fields of the JSON.

Documentation:
http://flink.incubator.apache.org/docs/0.7-incubating/streaming_guide.html#twitter-streaming-api

Relevant example code:
https://github.com/apache/incubator-flink/blob/master/flink-addons/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterLocal.java

Operator parsing the JSON into a DataStream:
https://github.com/apache/incubator-flink/blob/master/flink-addons/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/json/JSONParseFlatMap.java

Cheers,

Marton

On Tue, Oct 28, 2014 at 8:47 PM, Mustafa Elbehery <elbeherymustafa@gmail.com
> wrote:

> Hi All,
>
> We have created a Java model for the Twitter API, and are looking for a
> way to integrate it with Flink.
>
> *aalexandrov <https://dimatub.slack.com/team/aalexandrov>**[8:44 PM]*We
> have a sequence of tweets serialized as JSON objects and need to parse them
> accordingly in order to convert them to our Java model.
>
> *aalexandrov <https://dimatub.slack.com/team/aalexandrov>**[8:45 PM]*Is
> there an InputReader that can parse a JSON array into a sequence of strings
> (each string representing a single array element)?
> Regards.
>
> --
> Mustafa Elbehery
> EIT ICT Labs Master School <http://www.masterschool.eitictlabs.eu/home/>
> +49(0)16 2819 2148
> skype: mustafaelbehery87
>
>

Mime
View raw message