flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Flink and swapping question
Date Wed, 24 May 2017 12:32:12 GMT
Hi Greg,
I carefully monitored all TM memory with jstat -gcutil and there'no full
gc, only .
The initial situation on the dying TM is:

  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT
GCT
  0.00 100.00  33.57  88.74  98.42  97.17    159    2.508     1    0.255
 2.763
  0.00 100.00  90.14  88.80  98.67  97.17    197    2.617     1    0.255
 2.873
  0.00 100.00  27.00  88.82  98.75  97.17    234    2.730     1    0.255
 2.986

After about 10 hours of processing is:

  0.00 100.00  21.74  83.66  98.52  96.94   5519   33.011     1    0.255
33.267
  0.00 100.00  21.74  83.66  98.52  96.94   5519   33.011     1    0.255
33.267
  0.00 100.00  21.74  83.66  98.52  96.94   5519   33.011     1    0.255
33.267

So I don't think thta OOM could be an option.

However, the cluster is running on ESXi vSphere VMs and we already
experienced unexpected crash of jobs because of ESXi moving a heavy-loaded
VM to another (less loaded) physical machine..I would't be surprised if
swapping is also handled somehow differently..
Looking at Cloudera widgets I see that the crash is usually preceded by an
intense cpu_iowait period.
I fear that Flink unsafe access to memory could be a problem in those
scenarios. Am I wrong?

Any insight or debugging technique is  greatly appreciated.
Best,
Flavio


On Wed, May 24, 2017 at 2:11 PM, Greg Hogan <code@greghogan.com> wrote:

> Hi Flavio,
>
> Flink handles interrupts so the only silent killer I am aware of is
> Linux's OOM killer. Are you seeing such a message in dmesg?
>
> Greg
>
> On Wed, May 24, 2017 at 3:18 AM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
>> Hi to all,
>> I'd like to know whether memory swapping could cause a taskmanager crash.
>> In my cluster of virtual machines 'm seeing this strange behavior in my
>> Flink cluster: sometimes, if memory get swapped the taskmanager (on that
>> machine) dies unexpectedly without any log about the error.
>>
>> Is that possible or not?
>>
>> Best,
>> Flavio
>>
>
>

Mime
View raw message