flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zoudan <zoud...@163.com>
Subject Re: End to End Latency Tracking in flink
Date Thu, 02 Apr 2020 02:59:32 GMT
Hi,
I think we may add latency metric for each operator, which can reflect consumption ability
of each operator.

Best,
Dan Zou


> 在 2020年3月30日,18:19,Guanghui Zhang <begginghard@gmail.com> 写道:
> 
> Hi.
> At flink source connector, you can send $source_current_time - $event_time metric.
> In the meantime, at flink sink connector, you can send $sink_current_time - $event_time
metric.
> Then you use  $sink_current_time - $event_time - ($source_current_time - $event_time)
= $sink_current_time - $source_current_time as the latency of end to end。
> 
> Oscar Westra van Holthe - Kind <oscar@westravanholthe.nl <mailto:oscar@westravanholthe.nl>>
于2020年3月30日周一 下午5:15写道:
> On Mon, 30 Mar 2020 at 05:08, Lu Niu <qqibrow@gmail.com <mailto:qqibrow@gmail.com>>
wrote:
> $current_processing - $event_time works for event time. How about processing time? Is
there a good way to measure the latency? 
> 
> To measure latency you'll need some way to determine the time spent between the start
and end of your pipeline.
> 
> To measure latency when using processing time, you'll need to partially use ingestion
time. That is, you'll need to add the 'current' processing time as soon as messages are ingested.
> 
> With it, you can then use the $current_processing - $ingest_time solution that was already
mentioned.
> 
> Kind regards,
> Oscar
> 
> -- 
> Oscar Westra van Holthe - Kind


Mime
View raw message