Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANC1h_tT_pb=0y1txVM1sDo=8oiu5fiU2_cVfNECe0YB8oLpRA@mail.gmail.com>
References: <12E50A80607B5A46BF7060E19B29634F14DED7B4@NTOVMAIL04.ad.otto.de>
	<C6AEBAAC-8EA5-4EB8-B563-C921056ECA7C@ricobergmann.de>
	<CAAZ78qB+-FUhLLDA3j_XJ69dL2T6HTkuAMLu2exnG-gaQBGH0Q@mail.gmail.com>
	<7885A715-A8F3-4256-9A0B-7628BEF29675@ricobergmann.de>
	<CAGr9p8DzKgkfAft__xCHf_tkwzcpyv5+1LHwRc+hd2jYyBMOMw@mail.gmail.com>
	<84A966EF-C306-4FC0-9455-46445AC7656B@ricobergmann.de>
	<CAGr9p8A3A6yeDFpNRBTXvU=4pzqF_wtKiALoeuDMitg3PnFinQ@mail.gmail.com>
	<CD6C3334-205B-437D-9A1D-55C225A587EF@ricobergmann.de>
	<15DC7F60-EC46-455B-84E1-4E29DA852B45@ricobergmann.de>
	<CANMXwW3_UC3nki3QVgoHfP+Qt21XSDAucLVmkP2XMaB9TfpPhQ@mail.gmail.com>
	<02D694A0-FCE1-49BB-9026-5204EE3BCCA1@ricobergmann.de>
	<CANMXwW0r9hse1oAu__YiywMMpv3ESf6j4+1TmH7dg3SHq4Ns4Q@mail.gmail.com>
	<CANC1h_tT_pb=0y1txVM1sDo=8oiu5fiU2_cVfNECe0YB8oLpRA@mail.gmail.com>
Date: Wed, 9 Sep 2015 10:14:00 +0200
Message-ID: 
 <CADXjeyCEcpLF6yeb_OT1j=OJJs=FrNQTT9A+4SkVrEcZJNMidA@mail.gmail.com>
Subject: Re: Performance Issue
From: =?UTF-8?B?R8OhYm9yIEfDqXZheQ==?= <ggab90@gmail.com>
To: user@flink.apache.org
Content-Type: text/plain; charset=UTF-8

Btw, there was a discussion about this problem a while back:
https://mail-archives.apache.org/mod_mbox/flink-dev/201506.mbox/%3CCADXjeyCi9_OPRO4oQtzhvi-gifeK6_66YbtJZ_PB0aQOp_NijA@mail.gmail.com%3E
And here is the jira:
https://issues.apache.org/jira/browse/FLINK-2181

Best,
Gabor


2015-09-09 10:06 GMT+02:00 Stephan Ewen <sewen@apache.org>:
> Aljoscha and me are currently working on an alternative Windowing
> implementation. That new implementation will support out-of-order event time
> and release keys properly. We will hopefully have a first version to try out
> in a week or so...
>
> Greetings,
> Stephan
>
>
> On Wed, Sep 9, 2015 at 9:08 AM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>>
>> Ok, that's a special case but the system still shouldn't behave that way.
>> The problem is that the grouped discretizer that is responsible for grouping
>> the elements into grouped windows is keeping state for every key that it
>> encounters. And that state is never released, ever. That's the reason for
>> the hight memory consumption and GC load.
>>
>> On Wed, 9 Sep 2015 at 07:01 Rico Bergmann <info@ricobergmann.de> wrote:
>>>
>>> Yes. The keys are constantly changing. Indeed each unique event has its
>>> own key (the event itself). The purpose was to do an event deduplication ...
>>>
>>>
>>>
>>> Am 08.09.2015 um 20:05 schrieb Aljoscha Krettek <aljoscha@apache.org>:
>>>
>>> Hi Rico,
>>> I have a suspicion. What is the distribution of your keys? That is, are
>>> there many unique keys, do the keys keep evolving, i.e. is it always new and
>>> different keys?
>>>
>>> Cheers,
>>> Aljoscha
>>>
>>> On Tue, 8 Sep 2015 at 13:44 Rico Bergmann <info@ricobergmann.de> wrote:
>>>>
>>>> I also see in the TM overview the CPU load is still around 25% although
>>>> there is no input to the program since minutes. The CPU load is degrading
>>>> very slowly.
>>>>
>>>> The memory consumption is still fluctuating at a high level. It does not
>>>> degrade.
>>>>
>>>> In my test I generated test input for 1 minute. Now 10 minutes are over
>>>> ...
>>>>
>>>> I think there must be something with flink...
>>>>
>>>>
>>>>
>>>> Am 08.09.2015 um 13:32 schrieb Rico Bergmann <info@ricobergmann.de>:
>>>>
>>>> The marksweep value is very high, the scavenge very low. If this helps
>>>> ;-)
>>>>
>>>>
>>>>
>>>>
>>>> Am 08.09.2015 um 11:27 schrieb Robert Metzger <rmetzger@apache.org>:
>>>>
>>>> It is in the "Information" column: http://i.imgur.com/rzxxURR.png
>>>> In the screenshot, the two GCs only spend 84 and 25 ms.
>>>>
>>>> On Tue, Sep 8, 2015 at 10:34 AM, Rico Bergmann <info@ricobergmann.de>
>>>> wrote:
>>>>>
>>>>> Where can I find these information? I can see the memory usage and cpu
>>>>> load. But where are the information on the GC?
>>>>>
>>>>>
>>>>>
>>>>> Am 08.09.2015 um 09:34 schrieb Robert Metzger <rmetzger@apache.org>:
>>>>>
>>>>> The webinterface of Flink has a tab for the TaskManagers. There, you
>>>>> can also see how much time the JVM spend with garbage collection.
>>>>> Can you check whether the number of GC calls + the time spend goes up
>>>>> after 30 minutes?
>>>>>
>>>>> On Tue, Sep 8, 2015 at 8:37 AM, Rico Bergmann <info@ricobergmann.de>
>>>>> wrote:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I also think it's a GC problem. In the KeySelector I don't instantiate
>>>>>> any object. It's a simple toString method call.
>>>>>> In the mapWindow I create new objects. But I'm doing the same in other
>>>>>> map operators, too. They don't slow down the execution. Only with this
>>>>>> construct the execution is slowed down.
>>>>>>
>>>>>> I watched on the memory footprint of my program. Once with the code
>>>>>> construct I wrote and once without. The memory characteristic were the same.
>>>>>> The CPU usage also ...
>>>>>>
>>>>>> I don't have an explanation. But I don't think it comes from my
>>>>>> operator functions ...
>>>>>>
>>>>>> Cheers Rico.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am 07.09.2015 um 22:43 schrieb Martin Neumann <mneumann@sics.se>:
>>>>>>
>>>>>> Hej,
>>>>>>
>>>>>> This sounds like it could be a garbage collection problem. Do you
>>>>>> instantiate any classes inside any of the operators (e.g. in the
>>>>>> KeySelector). You can also try to run it locally and use something like
>>>>>> jstat to rule this out.
>>>>>>
>>>>>> cheers Martin
>>>>>>
>>>>>> On Mon, Sep 7, 2015 at 12:00 PM, Rico Bergmann <info@ricobergmann.de>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> While working with grouping and windowing I encountered a strange
>>>>>>> behavior. I'm doing:
>>>>>>>
>>>>>>> dataStream.groupBy(KeySelector).window(Time.of(x,
>>>>>>> TimeUnit.SECONDS)).mapWindow(toString).flatten()
>>>>>>>
>>>>>>>
>>>>>>> When I run the program containing this snippet it initially outputs
>>>>>>> data at a rate around 150 events per sec. (That is roughly the input rate
>>>>>>> for the program). After about 10-30 minutes the rate drops down below 5
>>>>>>> events per sec. This leads to event delivery offsets getting bigger and
>>>>>>> bigger ...
>>>>>>>
>>>>>>> Any explanation for this? I know you are reworking the streaming API.
>>>>>>> But it would be useful to know, why this happens ...
>>>>>>>
>>>>>>> Cheers. Rico.
>>>>>>
>>>>>>
>>>>>
>>>>
>