flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: [jira] [Commented] (FLINK-7606) CEP operator leaks state
Date Fri, 15 Sep 2017 15:54:22 GMT
Yes. In general, and for every keyed state, the amount of state you store
is proportional to the number of active keys that you expect to have.

On Sep 15, 2017 5:33 PM, "Paolo Rendano (JIRA)" <jira@apache.org> wrote:

>
>     [ https://issues.apache.org/jira/browse/FLINK-7606?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=16168025#comment-16168025 ]
>
> Paolo Rendano commented on FLINK-7606:
> --------------------------------------
>
> Hi [~kkl0u],
> from your comment I understand that the basic memory (so with minimal load
> of messages) I have to setup for a single instance is directly related with
> the number of keys I want to be able to manage. I correctly understood, do
> I?
>
> > CEP operator leaks state
> > ------------------------
> >
> >                 Key: FLINK-7606
> >                 URL: https://issues.apache.org/jira/browse/FLINK-7606
> >             Project: Flink
> >          Issue Type: Bug
> >          Components: CEP
> >    Affects Versions: 1.3.1
> >            Reporter: Matteo Ferrario
> >         Attachments: heap-dump1.png, heap-dump2.png, heap-dump3.png
> >
> >
> > The NestedMapsStateTable grows up continuously without free the heap
> memory.
> > We created a simple job that processes a stream of messages and uses CEP
> to generate an outcome message when a specific pattern is identified.
> > The messages coming from the stream are grouped by a key defined in a
> specific field of the message.
> > We've also added the "within" clause (set as 5 minutes), indicating that
> two incoming messages match the pattern only if they come in a certain time
> window.
> > What we've seen is that for every key present in the message, an NFA
> object is instantiated in the NestedMapsStateTable and it is never
> deallocated.
> > Also the "within" clause didn't help: we've seen that if we send
> messages that don't match the pattern, the memory grows up (I suppose that
> the state of NFA is updated) but it is not cleaned also after the 5 minutes
> of time window defined in "within" clause.
> > If you need, I can provide more details about the job we've implemented
> and also the screenshots about the memory leak.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message