flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arvid Heise <ar...@ververica.com>
Subject Re: Need suggestion on Flink-Kafka stream processing design
Date Tue, 12 May 2020 20:08:23 GMT
Hi Hemant,

In general, you want to keep all data coming from one device in one Kafka
partition, such that the timestamps of that device are monotonically
increasing. Thus, when processing data from one device, you have ensured
that no out-of-order events with respect to this device happen.

If you now want to aggregate all events of a given timestamp for a device,
it is a matter of keying by device id and applying a custom window. There
is no need for joins.

On Tue, May 12, 2020 at 9:05 PM hemant singh <hemant2184@gmail.com> wrote:

> Hello Flink Users,
>
> Any views on this question of mine.
>
> Thanks,
> Hemant
>
> On Tue, May 12, 2020 at 7:00 PM hemant singh <hemant2184@gmail.com> wrote:
>
>> Hello Roman,
>>
>> Thanks for your response.
>>
>> I think partitioning you described (event type + protocol type) is
>> subject to data skew. Including a device ID should solve this problem.
>> Also, including "protocol_type" into the key and having topic per
>> protocol_type seems redundant.
>> Each protocol is in single topic and event_type is key to distribute data
>> to a specific partition.
>>
>> Furthermore, do you have any particular reason to maintain multiple
>> topics?
>> I could imagine protocols have different speeds or other characteristics,
>> so you can tune Flink accordingly.
>> Otherwise, having a single topic partitioned only by device ID would
>> simplify deployment and reduce data skew.
>> Yes, you are right. These protocols have separate characteristics like
>> speed, data format. If I do have only one topic with data partitioned by
>> device_id then it could be that events from faster protocol is processed
>> faster and the joins which I want to do will not have enough matching data.
>> I have a question here how are you referring to tune Flink to handle
>> different characteristics like speed of streams as reading from kafka could
>> result in uneven processing of data?
>>
>> > By consume do you mean the downstream system?
>> My downstream is TSDB and other DBs where the data will be written to.
>> All these is time-series data.
>>
>> Thanks,
>> Hemant
>>
>>
>>
>> On Tue, May 12, 2020 at 5:28 PM Khachatryan Roman <
>> khachatryan.roman@gmail.com> wrote:
>>
>>> Hello Hemant,
>>>
>>> Thanks for your reply.
>>>
>>> I think partitioning you described (event type + protocol type) is
>>> subject to data skew. Including a device ID should solve this problem.
>>> Also, including "protocol_type" into the key and having topic per
>>> protocol_type seems redundant.
>>>
>>> Furthermore, do you have any particular reason to maintain multiple
>>> topics?
>>> I could imagine protocols have different speeds or other
>>> characteristics, so you can tune Flink accordingly.
>>> Otherwise, having a single topic partitioned only by device ID would
>>> simplify deployment and reduce data skew.
>>>
>>> > By consume do you mean the downstream system?
>>> Yes.
>>>
>>> Regards,
>>> Roman
>>>
>>>
>>> On Mon, May 11, 2020 at 11:30 PM hemant singh <hemant2184@gmail.com>
>>> wrote:
>>>
>>>> Hello Roman,
>>>>
>>>> PFB my response -
>>>>
>>>> As I understand, each protocol has a distinct set of event types (where
>>>> event type == metrics type); and a distinct set of devices. Is this correct?
>>>> Yes, correct. distinct events and devices. Each device emits these
>>>> event.
>>>>
>>>> > Based on data protocol I have 4-5 topics. Currently the data for a
>>>> single event is being pushed to a partition of the kafka topic(producer key
>>>> -> event_type + data_protocol).
>>>> Here you are talking about the source (to Flink job), right?
>>>> Yes, you are right.
>>>>
>>>> Can you also share how are you going to consume these data?
>>>> By consume do you mean the downstream system?
>>>> If yes then this data will be written to a DB, some metrics goes to
>>>> TSDB(Influx) as well.
>>>>
>>>> Thanks,
>>>> Hemant
>>>>
>>>> On Tue, May 12, 2020 at 2:08 AM Khachatryan Roman <
>>>> khachatryan.roman@gmail.com> wrote:
>>>>
>>>>> Hi Hemant,
>>>>>
>>>>> As I understand, each protocol has a distinct set of event types
>>>>> (where event type == metrics type); and a distinct set of devices. Is
this
>>>>> correct?
>>>>>
>>>>> > Based on data protocol I have 4-5 topics. Currently the data for
a
>>>>> single event is being pushed to a partition of the kafka topic(producer
key
>>>>> -> event_type + data_protocol).
>>>>> Here you are talking about the source (to Flink job), right?
>>>>>
>>>>> Can you also share how are you going to consume these data?
>>>>>
>>>>>
>>>>> Regards,
>>>>> Roman
>>>>>
>>>>>
>>>>> On Mon, May 11, 2020 at 8:57 PM hemant singh <hemant2184@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have different events from a device which constitutes different
>>>>>> metrics for same device. Each of these event is produced by the device
in
>>>>>> interval of few milli seconds to a minute.
>>>>>>
>>>>>> Event1(Device1) -> Stream1 -> Metric 1
>>>>>> Event2 (Device1) -> Stream2 -> Metric 2 ...
>>>>>> ..............
>>>>>> .......
>>>>>> Event100(Device1) -> Stream100 -> Metric100
>>>>>>
>>>>>> The number of events can go up to few 100s for each data protocol
and
>>>>>> we have around 4-5 data protocols. Metrics from different streams
makes up
>>>>>> a records
>>>>>> like for example from above example for device 1 -
>>>>>>
>>>>>> Device1 -> Metric1, Metric 2, Metric15 forms a single record for
the
>>>>>> device. Currently in development phase I am using interval join to
achieve
>>>>>> this, that is to create a record with latest data from different
>>>>>> streams(events).
>>>>>>
>>>>>> Based on data protocol I have 4-5 topics. Currently the data for
a
>>>>>> single event is being pushed to a partition of the kafka topic(producer
key
>>>>>> -> event_type + data_protocol). So essentially one topic is made
up of many
>>>>>> streams. I am filtering on the key to define the streams.
>>>>>>
>>>>>> My question is - Is this correct way to stream the data, I had
>>>>>> thought of maintaining different topic for an event, however in that
case
>>>>>> number of topics could go to few thousands and that is something
which
>>>>>> becomes little challenging to maintain and not sure if kafka handles
that
>>>>>> well.
>>>>>>
>>>>>> I know there are traditional ways to do this like pushing it to
>>>>>> timeseries db and then joining data for different metric but that
is
>>>>>> something which will never scale, also this processing should be
as
>>>>>> realtime as possible.
>>>>>>
>>>>>> Are there better ways to handle this use case or I am on correct
path.
>>>>>>
>>>>>> Thanks,
>>>>>> Hemant
>>>>>>
>>>>>

-- 

Arvid Heise | Senior Java Developer

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Mime
View raw message