flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourav Mazumder <sourav.mazumde...@gmail.com>
Subject Re: Streaming in Flink
Date Tue, 05 Jan 2016 06:01:34 GMT
Hi Gordon,

Need little more clarification around reading data from Kafka.

As soon as any component behaves as a consumer of a topic/queue (iof a
messaging system), it essentially does polling of the data after a regular
interval (that interval may be small though). Hence essentially it captures
all data/events accumulated in the queue from last polling instant to the
current polling in a pull manner

This pattern is very different from real time push of data where a daemon
process keeps on waiting on a continuous basis for any data pushed to it.

So what I'm looking clarification for is whether Flink supports a mechanism
where a data source (actually any client application) can push a data to a
socket which is continuously listened by a deamon process of flink.

Regards,
Sourav

On Mon, Jan 4, 2016 at 9:39 PM, Gordon Tai (戴資力) <tzulitai@gmail.com> wrote:

> Hi Sourav,
>
> Flink's streaming processes incoming data by-each-entry (true streaming,
> as compared to micro-batch), and streaming is inherently designed as a
> push-model, where a topology of stream transformations "listens" to a data
> source.
>
> You can have a Flink streaming topology's data source configured to be
> sockets or message queues such as Kafka's topics.
> Any event / data that is streamed to (or in other words, "pushed" to) the
> socket or Kafka topic will be processed by the Flink topology in real-time.
>
> Therefore, the answer is yes to your question. Hope this helps.
>
> BR,
> Gordon
>
> On Tue, Jan 5, 2016 at 12:42 PM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
>
>> Hi,
>>
>> Does Flink support push based data streaming where the data source can
>> push the events/data to Flink cluster over a socket (instead of Flink
>> pulling the data at a given frequency)?
>>
>> Regards,
>> Sourav
>>
>
>
>
> --
> *Tzu-Li Tai (Gordon Tai)*
> 戴資力
>
> *National Cheng Kung University, Graduate Institute of Computer and
> Communication Engineering*
> *High Performance Parallel and Distributed Systems Laboratory (HPDS Lab)*
> 國立成功大學電機工程學系 - 電腦與通信工程研究所
> 高效能平行/分散系統實驗室 (HPDS Lab)
>
> *National Cheng Kung University, Engineering Science Dpt.*
> 國立成功大學工程科學系
>
> *Contacts*
> tzulitai@ee.ncku.edu.tw
> http://tzulitai.ee.ncku.edu.tw
> Linkedin: tw.linkedin.com/in/tzulitai
> +886981916890
>

Mime
View raw message