Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Elias Levy <fearsome.lucidity@gmail.com>
Date: Thu, 4 May 2017 11:25:36 -0700
Message-ID: <CAFDmHdOB6k53FKvVXu78icH82evPUYyz=jEMbVk6MPQ59K72Jg@mail.gmail.com>
Subject: CEP memory requirements
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=f403045ffd286a7e6d054eb6e7ea
archived-at: Thu, 04 May 2017 18:26:08 -0000

--f403045ffd286a7e6d054eb6e7ea
Content-Type: text/plain; charset=UTF-8

I am observing odd memory behavior with the CEP library and I am wondering
if it is expected.

If I write a simple local streaming Flink job that reads from a 65MB
compressed file of JSON objects, one per line, parses the JSON, performs a
filter operation, and then a keyBy, heap usage is stable, staying below
250MB throughout per VisualVM.

But if I create a CEP pattern that matches nothing
(Pattern.begin[T]("foo").where( _ => false )) and match it against the
stream produced by the last keyBy (CEP.pattern(stream, pattern).select),
then memory balloons until the program terminates, steadily growing until
3GB.

The VisualVM memory profiler appears unable to account for that used heap
space.  If I add the Live Bytes column I'd get only between 200-100 MB.

Any idea what is going on?

Flink 1.2.  Java 8.

--f403045ffd286a7e6d054eb6e7ea
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I am observing odd memory behavior with the CEP library an=
d I am wondering if it is expected.<div><br></div><div>If I write a simple =
local streaming Flink job that reads from a 65MB compressed file of JSON ob=
jects, one per line, parses the JSON, performs a filter operation, and then=
 a keyBy, heap usage is stable, staying below 250MB throughout per VisualVM=
.</div><div><br></div><div>But if I create a CEP pattern that matches nothi=
ng (Pattern.begin[T](&quot;foo&quot;).where( _ =3D&gt; false )) and match i=
t against the stream produced by the last keyBy (CEP.pattern(stream, patter=
n).select), then memory balloons until the program terminates, steadily gro=
wing until 3GB.</div><div><br></div><div>The VisualVM memory profiler appea=
rs unable to account for that used heap space.=C2=A0 If I add the Live Byte=
s column I&#39;d get only between 200-100 MB.</div><div><br></div><div>Any =
idea what is going on?</div><div><br></div><div>Flink 1.2.=C2=A0 Java 8.</d=
iv><div><br></div></div>

--f403045ffd286a7e6d054eb6e7ea--