flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Flink UI records received/sent chained-unchained operators
Date Wed, 24 May 2017 15:48:51 GMT
I haven't been successful in reproducing this issue, nor did looking 
through the code provide any leads.

Does this happen every time? If so, could you enable the JMX reporter 
and check the numRecordsIn/-Out metrics for all operators/tasks?

On 24.05.2017 10:29, Chesnay Schepler wrote:
> Hello,
>
> I will try to reproduce this later today. My current assumption is 
> that the aggregation logic in the webUI for some reason adds the wrong 
> counters,
> it's unlikely that the counts are matching just by chance.
>
> Regards,
> Chesnay
>
> On 24.05.2017 10:15, Luis Lázaro wrote:
>> Hi everyone,
>> my use case:
>>
>> • Flink 1.3.0-SNAPSHOT
>> • DataStream API.
>> • Cluster mode is standalone, 3 TM and a JM.
>> • Parallelism is set to 1 on client level from UI.
>> • Source is Kafka configured with one single partition.
>>
>>  In Flink’s UI counter, I am getting a record count “sent" greater 
>> than the count of entries “received".Despite this, _total records is 
>> correct, ie, neither I losing nor am I getting more records._ Please 
>> check out screenshot#1:
>>
>>
>>
>>  My question is: *why the total number of records sent minus received 
>> is exactly the total leaked?* (250-223 = 27), i expected to see 27 
>> "Records sent”
>> ¿ Is Map->Filter-Filter, as a single operator a /SplitStream /?
>>
>> /.split(
>> case false => 223
>> case true => 27
>> ) /
>>
>> If I run _another job_ with the same execution plan but unchained 
>> operators in global operator (MAP->FILTER->FILTER) , the account is 
>> exactly the same records sent to the account filtered records. Please 
>> check out screenshot#2
>>
>>
>>
>>
>> PD: There is a similar question in Flink UI and records sent 
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-UI-and-records-sent-td12522.html>
, 
>> but I think they are not related.
>>
>> Thanks in advance, any suggestion is well appreciated.
>>
>> Best regards, Luis
>>
>>
>>
>


Mime
View raw message