Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 56A3E200C8C for ; Tue, 6 Jun 2017 19:42:21 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 54EC6160BC6; Tue, 6 Jun 2017 17:42:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 00FF4160BB7 for ; Tue, 6 Jun 2017 19:42:19 +0200 (CEST) Received: (qmail 10617 invoked by uid 500); 6 Jun 2017 17:42:18 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 10587 invoked by uid 99); 6 Jun 2017 17:42:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jun 2017 17:42:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 3C54B180647 for ; Tue, 6 Jun 2017 17:42:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.481 X-Spam-Level: ** X-Spam-Status: No, score=2.481 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=okkam-it.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id KZdcEv7OfPOs for ; Tue, 6 Jun 2017 17:42:13 +0000 (UTC) Received: from mail-ua0-f180.google.com (mail-ua0-f180.google.com [209.85.217.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 82B565F5B7 for ; Tue, 6 Jun 2017 17:42:13 +0000 (UTC) Received: by mail-ua0-f180.google.com with SMTP id h39so41842975uaa.3 for ; Tue, 06 Jun 2017 10:42:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=okkam-it.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Nlv8gdfFpseoPFe1ed16WJ+haHUPQnFw8ZCMVsMQv78=; b=OGUsdV6z0JpfEV5TnOwBbShKAgYTjhiyHAF2Y4Oj6YyAb+66omewZo9gIcXn+SssOZ VYaxSoVbmwyHhdch5yT7lu9a9ZDYVUYwrZlB3iS0Klm0rENvx62PWv5rWT49LFZMGO1L JocXSPrj/i2cg+Vtvs4tPcKzM8JJgZEHNHLtYi9tBLTmGRGRU7hvO4oKLBtQTmD7ye8q hebD2hWjyEP3LZ9YawQCbLxW6jnfqJ7Upec9LTBuEB/HXLnSGyzIF7cqkmY8NyXTyBH0 eDbGzMBBJPSnRQx2RdaU7WTaCTvVH2es/Zw9QGUtju+5lh0zm7aboQ2XjPmNLWj0SDmR SZjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Nlv8gdfFpseoPFe1ed16WJ+haHUPQnFw8ZCMVsMQv78=; b=euY9jTE+GeG/CqavyvIJmyZvY+rUtOKB9Fvfa+jv1yX9PkhwJcTRQ3EzQhdiZ59ZOT JAAAahm3Fz7wF1bO10EcGCm5A1JGjDkIOJ1cI5zIZvSDw7Y+0G2hIowL2LHLSE+GpOun WxlvHmPr+GfQKmPZGvIvEfyFzO+e0EJ07z5TW6caHATDy+DIEu4drATC7qrrNjHyfHaT yXWeLTAtK/nUZzqHtscP2+o5IvCVC9aVOEavyowEVr2rJowsJ6p2TP2jRRjas4fgn0VH UcfLDLQ2Waf79k34Hkar2VboNPZrayX5rx+3Tmj5OnkkTwD7Ae0TsMMe1YznLbPnf2+3 pbhQ== X-Gm-Message-State: AODbwcC67hD5hBSMtyfe7QA/95zs6m9gAyiJLariuTmrMPA8WLjv6szr 01Z0AP5aHynjvuLZ23u7j/2inLqYBHwF X-Received: by 10.176.67.135 with SMTP id l7mr15166982ual.143.1496770932839; Tue, 06 Jun 2017 10:42:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.52.145 with HTTP; Tue, 6 Jun 2017 10:42:12 -0700 (PDT) X-Originating-IP: [87.13.72.111] Received: by 10.31.52.145 with HTTP; Tue, 6 Jun 2017 10:42:12 -0700 (PDT) In-Reply-To: References: <71247100.Bo8u7n7Q2U@nico-work> From: Flavio Pompermaier Date: Tue, 6 Jun 2017 19:42:12 +0200 Message-ID: Subject: Re: Flink and swapping question To: Stephan Ewen Cc: Greg Hogan , Aljoscha Krettek , Fabian Hueske , user , Nico Kruber Content-Type: multipart/alternative; boundary="94eb2c083e22c3321405514e23ee" archived-at: Tue, 06 Jun 2017 17:42:21 -0000 --94eb2c083e22c3321405514e23ee Content-Type: text/plain; charset="UTF-8" Hi Stephan, I also think that the error is more related to netty. The only suspicious library I use are parquet or thrift. I'm not using off-heap memory. What do you mean for "crazy high number of concurrent network shuffles"?how can I count that? We're using java 8. Thanks a lot, Flavio On 6 Jun 2017 7:13 pm, "Stephan Ewen" wrote: Hi! I would actually be surprised if this is an issue in core Flink. - The MaxDirectMemory parameter is pretty meaningless, it really is a max and does not have an impact on how much is actually allocated. - In most cases we had reported so far, the leak was in a library that was used in the user code - If you do not use offheap memory in Flink, then there are few other culprits that can cause high virtual memory consumption: - Netty, if you bumped the Netty version in a custom build - Flink's Netty, if the job has a crazy high number of concurrent network shuffles (we are talking 1000s here) - Some old Java versions have I/O memory leaks (I think some older Java 6 and Java 7 versions were affected) To diagnose that better: - Are these batch or streaming jobs? - If it is streaming, which state backend are you using? Stephan On Tue, Jun 6, 2017 at 12:00 PM, Fabian Hueske wrote: > Hi Flavio, > > can you post the all memory configuration parameters of your workers? > Did you investigate which whether the direct or heap memory grew? > > Thanks, Fabian > > 2017-05-29 20:53 GMT+02:00 Flavio Pompermaier : > >> Hi to all, >> I'm still trying to understand what's going on our production Flink >> cluster. >> The facts are: >> >> 1. The Flink cluster runs on 5 VMWare VMs managed by ESXi >> 2. On a specific job we have, without limiting the direct memory to 5g, >> the TM gets killed by the OS almost immediately because the memory required >> by the TM, at some point, becomes huge, like > 100 GB (others jobs seem to >> be less affected by the problem ) >> 3. Although the memory consumption is much better this way, the Flink TM >> memory continuously grow job after job (of this problematic type): we set >> TM max heap to 14 GB and the JVM required memory can be ~ 30 Gb. How is >> that possible? >> >> My fear is that there's some annoying memory leak / bad memory allocation >> in the Flink network level, but I can't have any evidence of this (except >> the fact that the vm which doesn't have a hdfs datanode underneath the >> Flink TM is the one with the biggest TM virtual memory consumption). >> >> Thanks for the help , >> Flavio >> >> On 29 May 2017 15:37, "Nico Kruber" wrote: >> >>> FYI: taskmanager.sh sets this parameter but also states the following: >>> >>> # Long.MAX_VALUE in TB: This is an upper bound, much less direct >>> memory will >>> be used >>> TM_MAX_OFFHEAP_SIZE="8388607T" >>> >>> >>> Nico >>> >>> On Monday, 29 May 2017 15:19:47 CEST Aljoscha Krettek wrote: >>> > Hi Flavio, >>> > >>> > Is this running on YARN or bare metal? Did you manage to find out >>> where this >>> > insanely large parameter is coming from? >>> > >>> > Best, >>> > Aljoscha >>> > >>> > > On 25. May 2017, at 19:36, Flavio Pompermaier >>> > > wrote: >>> > > >>> > > Hi to all, >>> > > I think we found the root cause of all the problems. Looking ad dmesg >>> > > there was a "crazy" total-vm size associated to the OOM error, a LOT >>> much >>> > > bigger than the TaskManager's available memory. In our case, the TM >>> had a >>> > > max heap of 14 GB while the dmsg error was reporting a required >>> amount of >>> > > memory in the order of 60 GB! >>> > > >>> > > [ 5331.992539] Out of memory: Kill process 24221 (java) score 937 or >>> > > sacrifice child [ 5331.992619] Killed process 24221 (java) >>> > > total-vm:64800680kB, anon-rss:31387544kB, file-rss:6064kB, >>> shmem-rss:0kB >>> > > >>> > > That wasn't definitively possible usin an ordinary JVM (and our TM >>> was >>> > > running without off-heap settings) so we've looked at the parameters >>> used >>> > > to run the TM JVM and indeed there was a reall huge amount of memory >>> > > given to MaxDirectMemorySize. With my big surprise Flink runs a TM >>> with >>> > > this parameter set to 8.388.607T..does it make any sense?? Is it >>> > > documented anywhere the importance of this parameter (and why it is >>> used >>> > > in non off-heap mode as well)? Is it related to network buffers? It >>> > > should also be documented that this parameter should be added to the >>> TM >>> > > heap when reserving memory to Flin (IMHO). >>> > > >>> > > I hope that this painful sessions of Flink troubleshooting could be >>> an >>> > > added value sooner or later.. >>> > > >>> > > Best, >>> > > Flavio >>> > > >>> > > On Thu, May 25, 2017 at 10:21 AM, Flavio Pompermaier < >>> pompermaier@okkam.it >>> > > > wrote: I can confirm that after >>> giving >>> > > less memory to the Flink TM the job was able to run successfully. >>> After >>> > > almost 2 weeks of pain, we summarize here our experience with Fink in >>> > > virtualized environments (such as VMWare ESXi): Disable the >>> > > virtualization "feature" that transfer a VM from a (heavy loaded) >>> > > physical machine to another one (to balance the resource consumption) >>> > > Check dmesg when a TM dies without logging anything (usually it goes >>> OOM >>> > > and the OS kills it but there you can find the log of this thing) >>> CentOS >>> > > 7 on ESXi seems to start swapping VERY early (in my case I see the OS >>> > > starting swapping also if there are 12 out of 32 GB of free memory)! >>> > > We're still investigating how this behavior could be fixed: the >>> problem >>> > > is that it's better not to disable swapping because otherwise VMWare >>> > > could start ballooning (that is definitely worse...). >>> > > >>> > > I hope this tips could save someone else's day.. >>> > > >>> > > Best, >>> > > Flavio >>> > > >>> > > On Wed, May 24, 2017 at 4:28 PM, Flavio Pompermaier < >>> pompermaier@okkam.it >>> > > > wrote: Hi Greg, you were right! After >>> > > typing dmsg I found "Out of memory: Kill process 13574 (java)". This >>> is >>> > > really strange because the JVM of the TM is very calm. >>> > > Moreover, there are 7 GB of memory available (out of 32) but somehow >>> the >>> > > OS decides to start swapping and, when it runs out of available swap >>> > > memory, the OS decides to kill the Flink TM :( >>> > > >>> > > Any idea of what's going on here? >>> > > >>> > > On Wed, May 24, 2017 at 2:32 PM, Flavio Pompermaier < >>> pompermaier@okkam.it >>> > > > wrote: Hi Greg, >>> > > I carefully monitored all TM memory with jstat -gcutil and there'no >>> full >>> > > gc, only .> >>> > > The initial situation on the dying TM is: >>> > > S0 S1 E O M CCS YGC YGCT FGC >>> FGCT >>> > > GCT 0.00 100.00 33.57 88.74 98.42 97.17 159 2.508 1 >>> > > 0.255 2.763 0.00 100.00 90.14 88.80 98.67 97.17 197 >>> 2.617 >>> > > 1 0.255 2.873 0.00 100.00 27.00 88.82 98.75 97.17 >>> 234 >>> > > 2.730 1 0.255 2.986> >>> > > After about 10 hours of processing is: >>> > > 0.00 100.00 21.74 83.66 98.52 96.94 5519 33.011 1 >>> 0.255 >>> > > 33.267 0.00 100.00 21.74 83.66 98.52 96.94 5519 33.011 >>> 1 >>> > > 0.255 33.267 0.00 100.00 21.74 83.66 98.52 96.94 5519 >>> 33.011 >>> > > 1 0.255 33.267> >>> > > So I don't think thta OOM could be an option. >>> > > >>> > > However, the cluster is running on ESXi vSphere VMs and we already >>> > > experienced unexpected crash of jobs because of ESXi moving a >>> > > heavy-loaded VM to another (less loaded) physical machine..I would't >>> be >>> > > surprised if swapping is also handled somehow differently.. Looking >>> at >>> > > Cloudera widgets I see that the crash is usually preceded by an >>> intense >>> > > cpu_iowait period. I fear that Flink unsafe access to memory could >>> be a >>> > > problem in those scenarios. Am I wrong? >>> > > >>> > > Any insight or debugging technique is greatly appreciated. >>> > > Best, >>> > > Flavio >>> > > >>> > > >>> > > On Wed, May 24, 2017 at 2:11 PM, Greg Hogan >> > > > wrote: Hi Flavio, >>> > > >>> > > Flink handles interrupts so the only silent killer I am aware of is >>> > > Linux's OOM killer. Are you seeing such a message in dmesg? >>> > > >>> > > Greg >>> > > >>> > > On Wed, May 24, 2017 at 3:18 AM, Flavio Pompermaier < >>> pompermaier@okkam.it >>> > > > wrote: Hi to all, >>> > > I'd like to know whether memory swapping could cause a taskmanager >>> crash. >>> > > In my cluster of virtual machines 'm seeing this strange behavior in >>> my >>> > > Flink cluster: sometimes, if memory get swapped the taskmanager (on >>> that >>> > > machine) dies unexpectedly without any log about the error. >>> > > >>> > > Is that possible or not? >>> > > >>> > > Best, >>> > > Flavio >>> >>> > --94eb2c083e22c3321405514e23ee Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Stephan,
I also think that= the error is more related to netty.
The only suspic= ious library I use are parquet or thrift.
I'm no= t using off-heap memory.
What do you mean for "= crazy high number of concurrent netw= ork shuffles"?how can I count that?
We're using java 8.

Thanks a lot,
Flavio


On= 6 Jun 2017 7:13 pm, "Stephan Ewen" <sewen@apache.org> wrote:
Hi!

I would actually b= e surprised if this is an issue in core Flink.

=C2= =A0 - The MaxDirectMemory parameter is pretty meaningless, it really is a m= ax and does not have an impact on how much is actually allocated.

=C2=A0 - In most cases we had reported so far, the leak was= in a library that was used in the user code

= =C2=A0 - If you do not use offheap memory in Flink, then there are few othe= r culprits that can cause high virtual memory consumption:
=C2=A0= =C2=A0 =C2=A0 - Netty, if you bumped the Netty version in a custom build
=C2=A0 =C2=A0 =C2=A0 - Flink's Netty, if the job has a crazy h= igh number of concurrent network shuffles (we are talking 1000s here)
=
=C2=A0 =C2=A0 =C2=A0 - Some old Java versions have I/O memory leaks (I= think some older Java 6 and Java 7 versions were affected)

<= /div>

To diagnose that better:

= =C2=A0 - Are these batch or streaming jobs?=C2=A0
=C2=A0 - If it = is streaming, which state backend are you using?

Stephan
=

On Tue,= Jun 6, 2017 at 12:00 PM, Fabian Hueske <fhueske@gmail.com> = wrote:
Hi Flav= io,

can you post the all memory configuration parameters of yo= ur workers?
Did you investigate which whether the direct or h= eap memory grew?

Thanks, Fabian

2017-05-29 20:53 GMT+02:00 Flavio Pompermaier <pompermaier@= okkam.it>:
Hi to all,
I'm still trying to understand what's= going on our production Flink cluster.
The facts ar= e:

1. The Flink cluster = runs on 5 VMWare VMs managed by ESXi
2. On a specifi= c =C2=A0job we have, without limiting the direct memory to 5g, the TM gets = killed by the OS almost immediately because the memory required by the TM, = at some point, becomes huge, like > 100 GB (others jobs seem to be less = affected by the problem )
3. Although the memory con= sumption is much better this way, the Flink TM memory continuously grow job= after job (of this problematic type): we set TM max heap to 14 GB and the = JVM required memory can be ~ 30 Gb. How is that possible?

My fear is that there's some annoying= memory leak / bad memory allocation in the Flink network level, but I can&= #39;t have any evidence of this (except the fact that the vm which doesn= 9;t have a hdfs datanode underneath the Flink TM is the one with the bigges= t TM virtual memory consumption).

Thanks for the help ,
Flavio

On = 29 May 2017 15:37, "Nico Kruber" <nico@data-artisans.com> wrote:
FYI: taskmanager.sh sets = this parameter but also states the following:

=C2=A0 # Long.MAX_VALUE in TB: This is an upper bound, much less direct mem= ory will
be used
=C2=A0 TM_MAX_OFFHEAP_SIZE=3D"8388607T"


Nico

On Monday, 29 May 2017 15:19:47 CEST Aljoscha Krettek wrote:
> Hi Flavio,
>
> Is this running on YARN or bare metal? Did you manage to find out wher= e this
> insanely large parameter is coming from?
>
> Best,
> Aljoscha
>
> > On 25. May 2017, at 19:36, Flavio Pompermaier <pompermaier@okkam.it>
> > wrote:
> >
> > Hi to all,
> > I think we found the root cause of all the problems. Looking ad d= mesg
> > there was a "crazy" total-vm size associated to the OOM= error, a LOT much
> > bigger than the TaskManager's available memory. In our case, = the TM had a
> > max heap of 14 GB while the dmsg error was reporting a required a= mount of
> > memory in the order of 60 GB!
> >
> > [ 5331.992539] Out of memory: Kill process 24221 (java) score 937= or
> > sacrifice child [ 5331.992619] Killed process 24221 (java)
> > total-vm:64800680kB, anon-rss:31387544kB, file-rss:6064kB, shmem-= rss:0kB
> >
> > That wasn't definitively possible usin an ordinary JVM (and o= ur TM was
> > running without off-heap settings) so we've looked at the par= ameters used
> > to run the TM JVM and indeed there was a reall huge amount of mem= ory
> > given to MaxDirectMemorySize. With my big surprise Flink runs a T= M with
> > this parameter set to 8.388.607T..does it make any sense?? Is it<= br> > > documented anywhere the importance of this parameter (and why it = is used
> > in non off-heap mode as well)? Is it related to network buffers? = It
> > should also be documented that this parameter should be added to = the TM
> > heap when reserving memory to Flin (IMHO).
> >
> > I hope that this painful sessions of Flink troubleshooting could = be an
> > added value sooner or later..
> >
> > Best,
> > Flavio
> >
> > On Thu, May 25, 2017 at 10:21 AM, Flavio Pompermaier <pompermaier@okkam.it=
> > <mailto:pompermaier@okkam.it>> wrote: I can confirm that after giving=
> > less memory to the Flink TM the job was able to run successfully.= After
> > almost 2 weeks of pain, we summarize here our experience with Fin= k in
> > virtualized environments (such as VMWare ESXi): Disable the
> > virtualization "feature" that transfer a VM from a (hea= vy loaded)
> > physical machine to another one (to balance the resource consumpt= ion)
> > Check dmesg when a TM dies without logging anything (usually it g= oes OOM
> > and the OS kills it but there you can find the log of this thing)= CentOS
> > 7 on ESXi seems to start swapping VERY early (in my case I see th= e OS
> > starting swapping also if there are 12 out of 32 GB of free memor= y)!
> > We're still investigating how this behavior could be fixed: t= he problem
> > is that it's better not to disable swapping because otherwise= VMWare
> > could start ballooning (that is definitely worse...).
> >
> > I hope this tips could save someone else's day..
> >
> > Best,
> > Flavio
> >
> > On Wed, May 24, 2017 at 4:28 PM, Flavio Pompermaier <pompermaier@okkam.it=
> > <mailto:pompermaier@okkam.it>> wrote: Hi Greg, you were right! After<= br> > > typing dmsg I found "Out of memory: Kill process 13574 (java= )". This is
> > really strange because the JVM of the TM is very calm.
> > Moreover, there are 7 GB of memory available (out of 32) but some= how the
> > OS decides to start swapping and, when it runs out of available s= wap
> > memory, the OS decides to kill the Flink TM :(
> >
> > Any idea of what's going on here?
> >
> > On Wed, May 24, 2017 at 2:32 PM, Flavio Pompermaier <pompermaier@okkam.it=
> > <mailto:pompermaier@okkam.it>> wrote: Hi Greg,
> > I carefully monitored all TM memory with jstat -gcutil and there&= #39;no full
> > gc, only .>
> > The initial situation on the dying TM is:
> >=C2=A0 =C2=A0S0=C2=A0 =C2=A0 =C2=A0S1=C2=A0 =C2=A0 =C2=A0E=C2=A0 = =C2=A0 =C2=A0 O=C2=A0 =C2=A0 =C2=A0 M=C2=A0 =C2=A0 =C2=A0CCS=C2=A0 =C2=A0 Y= GC=C2=A0 =C2=A0 =C2=A0YGCT=C2=A0 =C2=A0 FGC=C2=A0 =C2=A0 FGCT
> >=C2=A0 =C2=A0GCT 0.00 100.00=C2=A0 33.57=C2=A0 88.74=C2=A0 98.42= =C2=A0 97.17=C2=A0 =C2=A0 159=C2=A0 =C2=A0 2.508=C2=A0 =C2=A0 =C2=A01
> >=C2=A0 =C2=A00.255=C2=A0 =C2=A0 2.763 0.00 100.00=C2=A0 90.14=C2= =A0 88.80=C2=A0 98.67=C2=A0 97.17=C2=A0 =C2=A0 197=C2=A0 =C2=A0 2.617
> >=C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 0.255=C2=A0 =C2=A0 2.873 0.00 = 100.00=C2=A0 27.00=C2=A0 88.82=C2=A0 98.75=C2=A0 97.17=C2=A0 =C2=A0 234
> >=C2=A0 =C2=A0 2.730=C2=A0 =C2=A0 =C2=A01=C2=A0 =C2=A0 0.255=C2=A0 = =C2=A0 2.986>
> > After about 10 hours of processing is:
> >=C2=A0 =C2=A00.00 100.00=C2=A0 21.74=C2=A0 83.66=C2=A0 98.52=C2=A0= 96.94=C2=A0 =C2=A05519=C2=A0 =C2=A033.011=C2=A0 =C2=A0 =C2=A01=C2=A0 =C2= =A0 0.255
> >=C2=A0 =C2=A033.267 0.00 100.00=C2=A0 21.74=C2=A0 83.66=C2=A0 98.5= 2=C2=A0 96.94=C2=A0 =C2=A05519=C2=A0 =C2=A033.011=C2=A0 =C2=A0 =C2=A01
> >=C2=A0 =C2=A00.255=C2=A0 =C2=A033.267 0.00 100.00=C2=A0 21.74=C2= =A0 83.66=C2=A0 98.52=C2=A0 96.94=C2=A0 =C2=A05519=C2=A0 =C2=A033.011
> >=C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 0.255=C2=A0 =C2=A033.267> > > So I don't think thta OOM could be an option.
> >
> > However, the cluster is running on ESXi vSphere VMs and we alread= y
> > experienced unexpected crash of jobs because of ESXi moving a
> > heavy-loaded VM to another (less loaded) physical machine..I woul= d't be
> > surprised if swapping is also handled somehow differently.. Looki= ng at
> > Cloudera widgets I see that the crash is usually preceded by an i= ntense
> > cpu_iowait period. I fear that Flink unsafe access to memory coul= d be a
> > problem in those scenarios. Am I wrong?
> >
> > Any insight or debugging technique is=C2=A0 greatly appreciated.<= br> > > Best,
> > Flavio
> >
> >
> > On Wed, May 24, 2017 at 2:11 PM, Greg Hogan <code@greghogan.com
> > <mailto:code@greghogan.com>> wrote: Hi Flavio,
> >
> > Flink handles interrupts so the only silent killer I am aware of = is
> > Linux's OOM killer. Are you seeing such a message in dmesg? > >
> > Greg
> >
> > On Wed, May 24, 2017 at 3:18 AM, Flavio Pompermaier <pompermaier@okkam.it=
> > <mailto:pompermaier@okkam.it>> wrote: Hi to all,
> > I'd like to know whether memory swapping could cause a taskma= nager crash.
> > In my cluster of virtual machines 'm seeing this strange beha= vior in my
> > Flink cluster: sometimes, if memory get swapped the taskmanager (= on that
> > machine) dies unexpectedly without any log about the error.
> >
> > Is that possible or not?
> >
> > Best,
> > Flavio




--94eb2c083e22c3321405514e23ee--