flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Congxian Qiu <qcx978132...@gmail.com>
Subject Re: Interval Join Late Record Metrics
Date Wed, 11 Dec 2019 08:55:49 GMT
Hi Chris

>From the code[1], currently, IntervalJoin will ignore the late data
silently, maybe you can create an issue to track this.

[1]
https://github.com/apache/flink/blob/5c89d12849ea2aa332126b32808e363f12d436a2/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222
Best,
Congxian


Chris Gillespie <cgillespie@zendesk.com> 于2019年12月11日周三 上午8:09写道:

> Hello Flink users, first time poster here.
>
> I'm using an interval join in my Flink project, however I haven't found
> where late records get logged in metrics. Window Joins have
> "numLateRecordsDropped" implemented
> <https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java#L141>,
> but is there an equivalent within an interval join?
>
> My main use case is to track how often a record falls outside of the lower
> and upper bounds when trying to join two streams. Interval Join looks like
> it simply short circuits
> <https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222>when
> there is a late record? Maybe I am not understanding what defines a late
> record in this situation.
>
> Is there a good way to monitor when an interval join fails to join two
> streams? Currently I'm looking at the delta between two operator metrics,
> but it hasn't looked that reliable so far.
>
> Thanks,
> Chris Gillespie
>

Mime
View raw message