Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Govindarajan Srinivasaraghavan <govindraghvan@gmail.com>
Date: Sun, 20 Aug 2017 20:29:44 -0700
Message-ID: <CAP2Faz=bqkAvja+NaUoiFcOiTH2R7QiyY9_+Zj8Ajf_JdOLR_A@mail.gmail.com>
Subject: Memory Issue
To: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="f403045fb9fa43043905573b18ab"
archived-at: Mon, 21 Aug 2017 03:30:21 -0000

--f403045fb9fa43043905573b18ab
Content-Type: text/plain; charset="UTF-8"

Hi,

I have a pipeline running on flink which ingests around 6k messages per
second. Each message is around 1kb and it passes through various stages
like filter, 5 sec tumbling window per key etc.. and finally flatmap to
computation before sending it to kafka sink. The data is first ingested as
protocol buffers and then in subsequent operators they are converted into
POJO's.

There are lots objects created inside the user functions and some of them
are cached as well. I have been running this pipeline on 48 task slots
across 3 task manages with each one allocated with 22GB memory.

The issue I'm having is within a period of 10 hours, almost 19k young
generation GC have been run which is roughly every 2 seconds and GC time
taken value is more than 2 million. I have also enabled object reuse. Any
suggestions on how this issue could be resolved? Thanks.

Regards,
Govind

--f403045fb9fa43043905573b18ab
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I have a pipeline running on flink =
which ingests around 6k messages per second. Each message is around 1kb and=
 it passes through various stages like filter, 5 sec tumbling window per ke=
y etc.. and finally flatmap to computation before sending it to kafka sink.=
 The data is first ingested as protocol buffers and then in subsequent oper=
ators they are converted into POJO&#39;s.</div><div><br></div><div>There ar=
e lots objects created inside the user functions and some of them are cache=
d as well. I have been running this pipeline on 48 task slots across 3 task=
 manages with each one allocated with 22GB memory.</div><div><br></div><div=
>The issue I&#39;m having is within a period of 10 hours, almost 19k young =
generation GC have been run which is roughly every 2 seconds and GC time ta=
ken value is more than 2 million. I have also enabled object reuse. Any sug=
gestions on how this issue could be resolved? Thanks.</div><div><br></div><=
div>Regards,</div><div>Govind</div><div><br></div></div>

--f403045fb9fa43043905573b18ab--