Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CA+3U0Z=dGL39Qxm36RBEfCfpugxQ42PJzF2O3frfOPk45bqbkg@mail.gmail.com>
References: <CAELUF_Ct+qvEc1BxuWg6wDP3u3XpXarh6q6ueS-vg0Yuh0QQqg@mail.gmail.com>
 <CA+3U0Z=dGL39Qxm36RBEfCfpugxQ42PJzF2O3frfOPk45bqbkg@mail.gmail.com>
From: Flavio Pompermaier <pompermaier@okkam.it>
Date: Wed, 24 May 2017 14:32:12 +0200
Message-ID: <CAELUF_Bi7Gpy2h54x1H-oTHcKugTJ6sAwCj6=GTXwS-YFTmcwA@mail.gmail.com>
Subject: Re: Flink and swapping question
To: Greg Hogan <code@greghogan.com>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="001a11425af85c6f940550444c8d"
archived-at: Wed, 24 May 2017 12:32:39 -0000

--001a11425af85c6f940550444c8d
Content-Type: text/plain; charset="UTF-8"

Hi Greg,
I carefully monitored all TM memory with jstat -gcutil and there'no full
gc, only .
The initial situation on the dying TM is:

  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT
GCT
  0.00 100.00  33.57  88.74  98.42  97.17    159    2.508     1    0.255
 2.763
  0.00 100.00  90.14  88.80  98.67  97.17    197    2.617     1    0.255
 2.873
  0.00 100.00  27.00  88.82  98.75  97.17    234    2.730     1    0.255
 2.986

After about 10 hours of processing is:

  0.00 100.00  21.74  83.66  98.52  96.94   5519   33.011     1    0.255
33.267
  0.00 100.00  21.74  83.66  98.52  96.94   5519   33.011     1    0.255
33.267
  0.00 100.00  21.74  83.66  98.52  96.94   5519   33.011     1    0.255
33.267

So I don't think thta OOM could be an option.

However, the cluster is running on ESXi vSphere VMs and we already
experienced unexpected crash of jobs because of ESXi moving a heavy-loaded
VM to another (less loaded) physical machine..I would't be surprised if
swapping is also handled somehow differently..
Looking at Cloudera widgets I see that the crash is usually preceded by an
intense cpu_iowait period.
I fear that Flink unsafe access to memory could be a problem in those
scenarios. Am I wrong?

Any insight or debugging technique is  greatly appreciated.
Best,
Flavio


On Wed, May 24, 2017 at 2:11 PM, Greg Hogan <code@greghogan.com> wrote:

> Hi Flavio,
>
> Flink handles interrupts so the only silent killer I am aware of is
> Linux's OOM killer. Are you seeing such a message in dmesg?
>
> Greg
>
> On Wed, May 24, 2017 at 3:18 AM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
>> Hi to all,
>> I'd like to know whether memory swapping could cause a taskmanager crash.
>> In my cluster of virtual machines 'm seeing this strange behavior in my
>> Flink cluster: sometimes, if memory get swapped the taskmanager (on that
>> machine) dies unexpectedly without any log about the error.
>>
>> Is that possible or not?
>>
>> Best,
>> Flavio
>>
>
>

--001a11425af85c6f940550444c8d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Greg,<div>I carefully monitored all TM memory with jsta=
t -gcutil and there&#39;no full gc, only .<div>The initial situation on the=
 dying TM is:</div><div><div><br></div><div>=C2=A0 S0 =C2=A0 =C2=A0 S1 =C2=
=A0 =C2=A0 E =C2=A0 =C2=A0 =C2=A0O =C2=A0 =C2=A0 =C2=A0M =C2=A0 =C2=A0 CCS =
=C2=A0 =C2=A0YGC =C2=A0 =C2=A0 YGCT =C2=A0 =C2=A0FGC =C2=A0 =C2=A0FGCT =C2=
=A0 =C2=A0 GCT =C2=A0=C2=A0</div><div>=C2=A0 0.00 100.00 =C2=A033.57 =C2=A0=
88.74 =C2=A098.42 =C2=A097.17 =C2=A0 =C2=A0159 =C2=A0 =C2=A02.508 =C2=A0 =
=C2=A0 1 =C2=A0 =C2=A00.255 =C2=A0 =C2=A02.763</div><div>=C2=A0 0.00 100.00=
 =C2=A090.14 =C2=A088.80 =C2=A098.67 =C2=A097.17 =C2=A0 =C2=A0197 =C2=A0 =
=C2=A02.617 =C2=A0 =C2=A0 1 =C2=A0 =C2=A00.255 =C2=A0 =C2=A02.873</div><div=
>=C2=A0 0.00 100.00 =C2=A027.00 =C2=A088.82 =C2=A098.75 =C2=A097.17 =C2=A0 =
=C2=A0234 =C2=A0 =C2=A02.730 =C2=A0 =C2=A0 1 =C2=A0 =C2=A00.255 =C2=A0 =C2=
=A02.986</div></div><div><br></div><div>After about 10 hours of processing =
is:<br></div><div><br></div><div><div>=C2=A0 0.00 100.00 =C2=A021.74 =C2=A0=
83.66 =C2=A098.52 =C2=A096.94 =C2=A0 5519 =C2=A0 33.011 =C2=A0 =C2=A0 1 =C2=
=A0 =C2=A00.255 =C2=A0 33.267</div><div>=C2=A0 0.00 100.00 =C2=A021.74 =C2=
=A083.66 =C2=A098.52 =C2=A096.94 =C2=A0 5519 =C2=A0 33.011 =C2=A0 =C2=A0 1 =
=C2=A0 =C2=A00.255 =C2=A0 33.267</div><div>=C2=A0 0.00 100.00 =C2=A021.74 =
=C2=A083.66 =C2=A098.52 =C2=A096.94 =C2=A0 5519 =C2=A0 33.011 =C2=A0 =C2=A0=
 1 =C2=A0 =C2=A00.255 =C2=A0 33.267</div></div><div><br></div><div>So I don=
&#39;t think thta OOM could be an option.</div><div><br></div><div>However,=
 the cluster is running on ESXi vSphere VMs and we already experienced unex=
pected crash of jobs because of ESXi moving a heavy-loaded VM to another (l=
ess loaded)=C2=A0physical machine..I would&#39;t be surprised if swapping i=
s also handled somehow differently..</div><div>Looking at Cloudera widgets =
I see that the crash is usually preceded by an intense cpu_iowait period.</=
div><div>I fear that Flink unsafe access to memory could be a problem in th=
ose scenarios. Am I wrong?</div><div><br></div><div>Any insight or debuggin=
g technique is =C2=A0greatly appreciated.</div><div>Best,</div><div>Flavio<=
/div><div><br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quot=
e">On Wed, May 24, 2017 at 2:11 PM, Greg Hogan <span dir=3D"ltr">&lt;<a hre=
f=3D"mailto:code@greghogan.com" target=3D"_blank">code@greghogan.com</a>&gt=
;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Hi =
Flavio,<br><br></div><div>Flink handles interrupts so the only silent kille=
r I am aware of is Linux&#39;s OOM killer. Are you seeing such a message in=
 dmesg?<span class=3D"HOEnZb"><font color=3D"#888888"><br></font></span></d=
iv><span class=3D"HOEnZb"><font color=3D"#888888"><div><br></div>Greg<br></=
font></span></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gma=
il_extra"><br><div class=3D"gmail_quote">On Wed, May 24, 2017 at 3:18 AM, F=
lavio Pompermaier <span dir=3D"ltr">&lt;<a href=3D"mailto:pompermaier@okkam=
.it" target=3D"_blank">pompermaier@okkam.it</a>&gt;</span> wrote:<br><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex"><div dir=3D"ltr">Hi to all,<div>I&#39;d like to kn=
ow whether memory swapping could cause a taskmanager crash.=C2=A0</div><div=
>In my cluster of virtual machines &#39;m seeing this strange behavior in m=
y Flink cluster: sometimes, if memory get swapped the taskmanager (on that =
machine) dies unexpectedly without any log about the error.<br></div><div><=
div><br></div><div>Is that possible or not?</div><div><br></div><div>Best,<=
/div><div>Flavio</div>
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br>
</div></div></div>

--001a11425af85c6f940550444c8d--