flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: FlinkCEP latency/throughput
Date Wed, 17 May 2017 15:34:29 GMT
Hello Alfred,

As a first general remark, Flink was not optimized for multicore deployments 
but rather for distributed environments. This implies overheads (serialization, 
communication etc), when compared to libs optimized for multicores. So there
may be libraries that are better optimized for those settings if you are planning 
to use just a multicore machine.

Now for your suggestion:

> On May 16, 2017, at 6:03 PM, Sonex <alfredjenssen@gmail.com> wrote:
> 
> Hello everyone,
> 
> I am testing some patterns with FlinkCEP and I want to measure latency and
> throughput when using 1 or more processing cores. How can I do that ??
> 
> What I have done so far:
> Latency: Each time an event arrives I store the system time
> (System.currentTimeMillis). When flink calls the select function which means
> we have a full pattern match, again I take the system time. The difference
> of the system time taken from the first event of the complex event and the
> system time taken when the function is called is the latency for now.
> 

1) If you are using event time, then you are also accounting for internal buffering and 
ordering of the incoming events.
 
2) I am not sure if measuring the time between the arrival of each element, and when 
its matching pattern is emitted makes much sense. In a long pattern, the first element
in the matching pattern will wait inevitably longer than the last one, right?

> Throughput: I divide the total number of the events of the dataset by the
> time taken to complete the experiment.
> 
> 

For throughput you could create a job with a sink that does nothing and only a CEP pattern
in your job and count the elements read by your source/min. If your source is not the bottleneck
then the CEP part of the pipeline is the dominating factor (given that your sink just discards
everything
so it cannot create backpressure).

I hope this helps,
Kostas
Mime
View raw message