flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Walther <twal...@apache.org>
Subject Re: DataStream of Future
Date Fri, 16 Sep 2016 11:51:11 GMT
Hi Albert,

you cannot use Futures between operators as objects are seralialized and 
possibly sent through the cluster immediatly. Right now there is no 
straight forward way in Flink to do async calls. However, there is a 
discussion going on which you might wanna join [1]. As far as I know, 
the current solution is to create a FlatMap function manually which 
manages the async calls and emits [2].

[1] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Proposal-for-Asynchronous-I-O-in-FLINK-td13497.html
[2] 
http://stackoverflow.com/questions/38866078/how-to-look-up-and-update-the-state-of-a-record-from-a-database-in-apache-flink

I hope that helps.

Timo



Am 16/09/16 um 13:16 schrieb Albert Gimenez:
> Hello,
>
>   
>
> I just started learning Flink (using Scala) recently, and I developed a job
> that, in short, does this steps:
>
>   
>
> -          Reads json messages from Kafka
>
> -          Enriches the messages, reading data from Cassandra (using Phantom
> DSL)
>
> -          Puts the enriched messages back to another Kafka topic.
>
>   
>
> The job looks like this:
>
>   
>
> env
>
>      .addSource(new FlinkKafkaProducer09[String](...))
>
>      .map(MyService.enrichMessage _) // Returns Option
>
>      .filter(!_.isEmpty)
>
>      .map(_.get)
>
>      .map(enrichedMessageToJsonMapper)
>
>      .addSink(new FlinkKafkaConsumer09[String](...)))
>
>   
>
> The "enrichMessage" method is where I'm using Phantom DSL to query
> Cassandra, and I would like to return a Future, but I can't figure out a way
> to do it right now, so I'm using "Await" to force the resolution and return
> a result. Is there a way to use a Future here?
>
>   
>
> I do have a second job that is updating the data in Cassandra, and since I
> don't need to sink, I can have my map to return the Futures, and everything
> happens asynchronously. I would like to know if it's possible to have a
> similar behavior when I want to use a Sink (so, sink to Kafka as the Futures
> are completed).
>
>   
>
> BTW, I'm using Flink 1.1.2 with Scala 2.11.
>
>   
>
> Thanks a lot for your help!
>
>   
>
> Kind regards,
>
>   
>
> Albert
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>


-- 
Freundliche Grüße / Kind Regards

Timo Walther

Follow me: @twalthr
https://www.linkedin.com/in/twalthr


Mime
View raw message