flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Flink UI records received/sent chained-unchained operators
Date Fri, 02 Jun 2017 12:49:46 GMT
Pretty sure i know what's going on. If I'm correct, giving the second 
Filter another name (by calling filter(...).name("superUniqueFilter")) 
should resolve the issue.

The metric system uses the operator names to distinguish operators, 
because Flink currently doesn't expose a unique ID for them. (They 
exist, but only in a very small part at the moment)

So what happens is that both filters count their output, and due to some 
internal technicalities this means that the output of the task is the 
output of both filters, added together.

Your first filter apparently isn't filtering anything (so it emits 4.529 
records); with the output of the second filter (803) we then arrive at 5332.

On 01.06.2017 16:14, Luis Lázaro wrote:
> Hi,
> i am reporting metrics from JMX and getting similar results:
>
>   * "Records received" < Records sent" if Operator  Map->
>     filter(condition1)->filter(condition2), as the snapshot.
>   *  if Operator is Map->filter(condition1&&condition2) results are
>     correct (Received sent = 803)
>   * Next operator Sink is receiving Previous_sent - Previous_Received
>     (5332 - 4529 = 803)
>
>
>
>
> Thanks in advance, any suggestion is well appreciated.
>
> Best regards, Luis
>
>
>
>


Mime
View raw message