flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rico Bergmann <i...@ricobergmann.de>
Subject Re: Performance Issue
Date Tue, 08 Sep 2015 06:37:51 GMT
Hi!

I also think it's a GC problem. In the KeySelector I don't instantiate any object. It's a
simple toString method call. 
In the mapWindow I create new objects. But I'm doing the same in other map operators, too.
They don't slow down the execution. Only with this construct the execution is slowed down.


I watched on the memory footprint of my program. Once with the code construct I wrote and
once without. The memory characteristic were the same. The CPU usage also ... 

I don't have an explanation. But I don't think it comes from my operator functions ...

Cheers Rico. 



> Am 07.09.2015 um 22:43 schrieb Martin Neumann <mneumann@sics.se>:
> 
> Hej,
> 
> This sounds like it could be a garbage collection problem. Do you instantiate any classes
inside any of the operators (e.g. in the KeySelector). You can also try to run it locally
and use something like jstat to rule this out.
> 
> cheers Martin
> 
>> On Mon, Sep 7, 2015 at 12:00 PM, Rico Bergmann <info@ricobergmann.de> wrote:
>> Hi!
>> 
>> While working with grouping and windowing I encountered a strange behavior. I'm doing:
>>> dataStream.groupBy(KeySelector).window(Time.of(x, TimeUnit.SECONDS)).mapWindow(toString).flatten()
>> 
>> When I run the program containing this snippet it initially outputs data at a rate
around 150 events per sec. (That is roughly the input rate for the program). After about 10-30
minutes the rate drops down below 5 events per sec. This leads to event delivery offsets getting
bigger and bigger ... 
>> 
>> Any explanation for this? I know you are reworking the streaming API. But it would
be useful to know, why this happens ...
>> 
>> Cheers. Rico. 
> 

Mime
View raw message