flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Gillespie <cgilles...@zendesk.com>
Subject Re: Interval Join Late Record Metrics
Date Wed, 11 Dec 2019 17:22:22 GMT
Thanks Congxian, I made a JIRA to track this request.
https://issues.apache.org/jira/browse/FLINK-15202

On Wed, Dec 11, 2019 at 12:56 AM Congxian Qiu <qcx978132955@gmail.com>
wrote:

> Hi Chris
>
> From the code[1], currently, IntervalJoin will ignore the late data
> silently, maybe you can create an issue to track this.
>
> [1]
> https://github.com/apache/flink/blob/5c89d12849ea2aa332126b32808e363f12d436a2/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222
> <https://github.com/apache/flink/blob/5c89d12849ea2aa332126b32808e363f12d436a2/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222>
> Best,
> Congxian
>
>
> Chris Gillespie <cgillespie@zendesk.com> 于2019年12月11日周三 上午8:09写道:
>
>> Hello Flink users, first time poster here.
>>
>> I'm using an interval join in my Flink project, however I haven't found
>> where late records get logged in metrics. Window Joins have
>> "numLateRecordsDropped" implemented
>> <https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java#L141>,
but is
>> there an equivalent within an interval join?
>>
>> My main use case is to track how often a record falls outside of the
>> lower and upper bounds when trying to join two streams. Interval Join looks
>> like it simply short circuits
>> <https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222>when
there is
>> a late record? Maybe I am not understanding what defines a late record in
>> this situation.
>>
>> Is there a good way to monitor when an interval join fails to join two
>> streams? Currently I'm looking at the delta between two operator metrics,
>> but it hasn't looked that reliable so far.
>>
>> Thanks,
>> Chris Gillespie
>>
>

Mime
View raw message