flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Rovner <alex.rov...@magnetic.com>
Subject streaming state
Date Fri, 11 Dec 2015 15:00:55 GMT
Hello all,

I was wondering if someone would be kind enough to enlighten me on a few
topics. We are trying to join two streams of data on a key. We were
thinking of partitioning topics in Kafka by the key, however I also saw
that Flink is able to partition on its own and I was wondering whether
Flink can take advantage of Kafka's partitioning and/or which partitioning
scheme should I go for?

As far as joins, our two datasets are very large (millions of records in
each window) and we need to perform the joins very quickly (less than 1
sec). Would the built in join mechanism be sufficient for this? From what I
understand it would need to shuffle data and thus be slow on large
datasets? I was wondering if there is a way to join via state key value
lookups to avoid the shuffling?

I read the docs and the blogs so far, thus have some limited understanding
of how Flink works, no practical experience though.


*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052

* <http://www.magnetic.com/>*

View raw message