Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6E6D4200B23 for ; Sun, 19 Jun 2016 18:16:54 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6CE46160A53; Sun, 19 Jun 2016 16:16:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8F487160A4E for ; Sun, 19 Jun 2016 18:16:53 +0200 (CEST) Received: (qmail 91737 invoked by uid 500); 19 Jun 2016 16:16:52 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 91720 invoked by uid 99); 19 Jun 2016 16:16:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2016 16:16:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 6A11CC0FE3 for ; Sun, 19 Jun 2016 16:16:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.493 X-Spam-Level: ** X-Spam-Status: No, score=2.493 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_IMAGE_RATIO_06=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eoZVkTdof9nZ for ; Sun, 19 Jun 2016 16:16:48 +0000 (UTC) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id D4C9E5F56F for ; Sun, 19 Jun 2016 16:16:47 +0000 (UTC) Received: by mail-wm0-f54.google.com with SMTP id f126so42145562wma.1 for ; Sun, 19 Jun 2016 09:16:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=sl6AtmfC9wYVN4agZwnlvzu3MQNlpr+QlTtYnK+HMUc=; b=mQZQIJDrIxCEb8ZwhpoWbXkL9M1/dxigU0X/w8OBusKKtubTY88i33rpYl7a0HnaIu 1+8w5Yo6sGjvfIm71vnzqayHsgaq6ki/OjkbS61CdWKdTFYHV8NsEgLXG1yAlOfBg/H5 h/OW4ofu0LBk2jrMdFMMjJsX7eOs9GIoQ2E4jNcf7E1hSa0JqEkSd7pE2mm0P+w0Y3rn vfHo39Y9PsWvGvgJvSc5qhr+44vEr0xSzBt+YYxUG3wl1i7NVp9LqPvb9tjRH2PLHNe6 DjpTn6pb7mFNtGiNSE1l9arwrJkuew9+tvzDZ0gHLNbDe6QpjfW7JPW+ldWlO62g8o7U kJpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=sl6AtmfC9wYVN4agZwnlvzu3MQNlpr+QlTtYnK+HMUc=; b=eKARwUauLEGglImD4nkbGgDiy0SADoBtV52vB8DfIOSCa1gW1shxokV2Dh17K66H4I ImAWeSwZ0to5gPbbssY3hYoW8p4acmTERlYEnr/EENgXOdcZz8v8llOOJxuigW7YEL5X /JgBSxBh6fJV1u3ScLg9u/wb2zeE0EwMyNKVNUpugwz+ivvpXXXEyu2qbd1UwNEi5sds 5gJis1KkE1FRxu6skro7kFBkfB0qqx98qdqW1Yq5hX9mdAa2egbLSttkHOdC5/SWMu2Y +uszWmPsEek2cCu2aGH3xaTkl/ylMy9gwivqYffFNqOtwymhCNeZeCQddE8lyeTRWoaL uf7Q== X-Gm-Message-State: ALyK8tLyLtf9KoCaIoMDJnj/vmA7sbUixAKyxFwuOFUy1XTcJo/dBCxIVO4jHTTX4Vx/1j+mEf53ac24ToVNVw== X-Received: by 10.28.134.199 with SMTP id i190mr7610063wmd.26.1466353007039; Sun, 19 Jun 2016 09:16:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.171.7 with HTTP; Sun, 19 Jun 2016 09:16:45 -0700 (PDT) In-Reply-To: References: From: CPC Date: Sun, 19 Jun 2016 19:16:45 +0300 Message-ID: Subject: Re: offheap memory allocation and memory leak bug To: dev@flink.apache.org Content-Type: multipart/related; boundary=001a114423381a80400535a3ea9a archived-at: Sun, 19 Jun 2016 16:16:54 -0000 --001a114423381a80400535a3ea9a Content-Type: multipart/alternative; boundary=001a114423381a803e0535a3ea99 --001a114423381a803e0535a3ea99 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, I think i found some information regarding this behavior. In jvm it is almost imposible to free allocated memory via ByteBuffer.allocateDirect. There is no explicit way to say jvm "free this direct bytebuffer". In some forums they said you can free memory with below method: > def releaseBuffers(buffers:List[ByteBuffer]):List[ByteBuffer] =3D { > > if(!buffers.isEmpty){ > > val cleanerMethod =3D buffers.head.getClass.getMethod("cleaner") > > cleanerMethod.setAccessible(true) > > buffers.foreach{buffer=3D> > > val cleaner =3D cleanerMethod.invoke(buffer) > > val cleanMethod =3D cleaner.getClass().getMethod("clean") > > cleanMethod.setAccessible(true) > > cleanMethod.invoke(cleaner) > > } > > } > > List.empty[ByteBuffer] > > } > > but since cleaner method is an internal method ,above is not recommended and not working in every jvm and java 9 does not support it also. I also made some tests with above method and behavior is not predictable. If memory allocated by some other thread and that thread exit then it release memory. Actually GC controls directMemory buffers. If there is no gc activity and memory is allocated and then dereferenced by different threads memory usage goes beyond intended and machine goes to swap then os kills taskmanager. In my tests i saw that behaviour: Suppose that thread A allocated 8gb memory exit and there is no reference to allocated memory than thread B allocated 8gb memory exit and there is no reference to allocated memory when i look at direct memory usage from jvisualvm it looks like below(-Xmx512m -XX:MaxDirectMemorySize=3D12G) [image: Inline images 1] but RSS of the process is 16 GB. If i call System.gc at that point RSS drops to 8GB but not to expected point. This is why Apache cassandra guys select sun.misc.Unsafe( http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Off-heap-c= aching-through-ByteBuffer-allocateDirect-when-JNA-not-available-td6977711.h= tml ). I think currently only way to limit memory usage in flink if you want to use same taskmanager across jobs is via "taskmanager.memory.preallocate: true". Since it allocate memory at the beginning and not freed its memory usage stays constant. PS: Sorry for my english i am not a native speaker. I hope i can explain what i intended to :) On 18 June 2016 at 16:36, CPC wrote: > Hello, > > I repeated the same test with conf values. > >> taskmanager.heap.mb: 6500 >> >> taskmanager.memory.off-heap: true >> >> taskmanager.memory.fraction: 0.9 >> >> > i set TM_MAX_OFFHEAP_SIZE=3D"6G" in taskmanager sh. Taskmanager started w= ith > >> capacman 14543 323 56.0 17014744 13731328 pts/1 Sl 16:23 35:25 >> /home/capacman/programlama/java/jdk1.7.0_75/bin/java >> -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xms650M -Xmx650M >> -XX:MaxDirectMemorySize=3D6G -XX:MaxPermSize=3D256m >> -Dlog.file=3D/home/capacman/Data/programlama/flink-1.0.3/log/flink-capac= man-taskmanager-0-capacman-Aspire-V3-771.log >> -Dlog4j.configuration=3Dfile:/home/capacman/Data/programlama/flink-1.0.3= /conf/log4j.properties >> -Dlogback.configurationFile=3Dfile:/home/capacman/Data/programlama/flink= -1.0.3/conf/logback.xml >> -classpath >> /home/capacman/Data/programlama/flink-1.0.3/lib/flink-dist_2.11-1.0.3.ja= r:/home/capacman/Data/programlama/flink-1.0.3/lib/flink-python_2.11-1.0.3.j= ar:/home/capacman/Data/programlama/flink-1.0.3/lib/log4j-1.2.17.jar:/home/c= apacman/Data/programlama/flink-1.0.3/lib/slf4j-log4j12-1.7.7.jar::: >> org.apache.flink.runtime.taskmanager.TaskManager --configDir >> /home/capacman/Data/programlama/flink-1.0.3/conf >> > > but memory usage reach up to 13Gb. Could somebodey explain me why memory > usage is so high? I expect it to be at most 8GB with some jvm internal > overhead. > > [image: Inline images 1] > > [image: Inline images 2] > > On 17 June 2016 at 20:26, CPC wrote: > >> Hi, >> >> I am making some test about offheap memory usage and encounter an odd >> behavior. My taskmanager heap limit is 12288 Mb and when i set >> "taskmanager.memory.off-hep:true" for every job it allocates 11673 Mb of= f >> heap area at most which is heapsize*0.95(value of >> taskmanager.memory.fraction). But when i submit second job it allocated >> another 11GB and does not free memory since MaxDirectMemorySize set to >> -XX:MaxDirectMemorySize=3D${TM_MAX_OFFHEAP_SIZE}" which is >> TM_MAX_OFFHEAP_SIZE=3D"8388607T" and my laptop goes to swap then kernel = oom >> killed taskmanager. If i hit perform gc from visualvm between jobs then = it >> release direct memory but memory usage of taskmanager in ps command is >> still around 20GB(RSS) and 27GB(virtual size) in that case i could subm= it >> my test job a few times without oom killed task manager but after 10 sub= mit >> it killed taskmanager again. I dont understand why jvm memory usage is >> still high even if all direct memory released. Do you have any idea? The= n >> i set MaxDirectMemorySize to 12 GB in this case it freed direct memory >> without any explicit gc triggering from visualvm but jvm process memory >> usage was still high around 20GB(RSS) and 27GB(virtual size). After agai= n >> maybe 10 submit it killed taskmanager. I think this is a bug and make it >> imposible to reuse taskmanagers without restarting them in standalone mo= de. >> >> [image: Inline images 1] >> >> [image: Inline images 2] >> > > --001a114423381a803e0535a3ea99 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

I think i found some information re= garding this behavior.=C2=A0 In jvm it is almost imposible to free allocate= d memory via ByteBuffer.allocateDirect. There is no explicit way to say jvm= "free this direct bytebuffer". In some forums they said you can = free memory with below method:=C2=A0
def releaseBuffers(buffers:List[ByteBuffer]):List[ByteBuffer] =3D {
=C2=A0 =C2=A0 if(!buffers.isEmpty){
=C2=A0 =C2=A0 =C2=A0 =C2=A0 val cleanerMethod =3D buffers.head= .getClass.getMethod("cleaner")
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 cleanerMethod.setAccessible(true)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 buffers.foreach{buffer=3D>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 val clea= ner =3D cleanerMethod.invoke(buffer)
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 val cleanMethod =3D cleaner.getClass().g= etMethod("clean")
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 cleanMethod.setAccessible(true)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 cleanMethod.invoke(cle= aner)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 }
=C2=A0 =C2= =A0 List.empty[ByteBuffer]
}

but since cleaner method is an internal method = ,above =C2=A0is not recommended and not working in every jvm and java 9 doe= s not support it also. I also made some tests with above method and behavio= r is not predictable. If memory allocated by some other thread and that thr= ead exit then it release memory. Actually GC controls directMemory buffers.= If there is no gc activity and memory is allocated and then dereferenced b= y different threads memory usage goes beyond intended and machine goes to s= wap then os kills taskmanager. In my tests i saw that behaviour:
=
Suppose that thread A allocated 8gb memory exit and there is= no reference to allocated memory
than thread B allocated 8gb mem= ory exit and there is no reference to allocated memory

=
when i look at direct memory usage from jvisualvm it looks like below(= -Xmx512m -XX:MaxDirectMemorySize=3D12G)

3D"Inline

but RSS of the process is 16 GB. If = i call System.gc at that point RSS drops to 8GB but not to expected point.<= /div>


I think cur= rently only way to limit memory usage in flink if you want to use same task= manager across jobs is via "taskmanager.memory.preallocate: true"= . Since it allocate memory at the beginning and not freed its memory usage = stays constant.

PS: Sorry for my english i am not = a native speaker. I hope i can explain what i intended to :)

=


On 18 June 2016 at 16:36, CPC <achalil@gmail.com> = wrote:
Hello,

I repeated the same test with conf values.
=
taskmanager.heap.mb: 6500
tas= kmanager.memory.off-heap: true
taskmanager.mem= ory.fraction: 0.9
=C2=A0
i set= =C2=A0TM_MAX_OFFHEAP_SIZE=3D"6G" in taskmanager sh. Taskmanager s= tarted with=C2=A0
capacman 14543 =C2=A0323 56.0 17014= 744 13731328 pts/1 Sl 16:23 =C2=A035:25 /home/capacman/programlama/java/jdk= 1.7.0_75/bin/java -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xm= s650M -Xmx650M -XX:MaxDirectMemorySize=3D6G -XX:MaxPermSize=3D256m -Dlog.fi= le=3D/home/capacman/Data/programlama/flink-1.0.3/log/flink-capacman-taskman= ager-0-capacman-Aspire-V3-771.log -Dlog4j.configuration=3Dfile:/home/capacm= an/Data/programlama/flink-1.0.3/conf/log4j.properties -Dlogback.configurati= onFile=3Dfile:/home/capacman/Data/programlama/flink-1.0.3/conf/logback.xml = -classpath /home/capacman/Data/programlama/flink-1.0.3/lib/flink-dist_2.11-= 1.0.3.jar:/home/capacman/Data/programlama/flink-1.0.3/lib/flink-python_2.11= -1.0.3.jar:/home/capacman/Data/programlama/flink-1.0.3/lib/log4j-1.2.17.jar= :/home/capacman/Data/programlama/flink-1.0.3/lib/slf4j-log4j12-1.7.7.jar:::= org.apache.flink.runtime.taskmanager.TaskManager --configDir /home/capacma= n/Data/programlama/flink-1.0.3/conf

but= memory usage reach up to 13Gb. Could somebodey explain me why memory usage= is so high? I expect it to be at most 8GB with some jvm internal overhead.= =C2=A0

3D"Inline
<= div>
3D"Inline

On 17 June 2016 at 20:26, CPC <achalil@gmail.com> wrote= :
Hi,

I am making some test about offheap memory usage and encounter an odd beha= vior. My taskmanager heap limit is 12288 Mb and when i set "taskmanage= r.memory.off-hep:true" for every job it allocates 11673 Mb off heap ar= ea at most which is heapsize*0.95(value of taskmanager.memory.fraction). Bu= t when i submit second job it allocated another 11GB and does not free memo= ry since MaxDirectMemorySize set to =C2=A0-XX:MaxDirectMemorySize=3D${TM_MA= X_OFFHEAP_SIZE}" =C2=A0which is TM_MAX_OFFHEAP_SIZE=3D"8388607T&q= uot; and my laptop goes to swap then kernel oom killed taskmanager. If i hi= t perform gc from visualvm between jobs then it release direct memory but m= emory usage of taskmanager in ps command is still around 20GB(RSS) and 27GB= (virtual size) =C2=A0in that case i could submit my test job a few times wi= thout oom killed task manager but after 10 submit =C2=A0it killed taskmanag= er again.=C2=A0 I dont understand why jvm memory usage is still high even i= f all direct memory released. Do you have any idea? Then =C2=A0i set=C2=A0M= axDirectMemorySize=C2=A0to 12 GB =C2=A0in this case it freed direct memory = without any explicit gc triggering from visualvm but jvm process memory usa= ge was still high around 20GB(RSS) and 27GB(virtual size). After again mayb= e 10 submit it killed taskmanager.=C2=A0I think this is a bug and=C2=A0make= it imposible to reuse taskmanagers without=C2=A0restarting them in standal= one mode.

=3D"Inline

=
3D"Inline


--001a114423381a803e0535a3ea99-- --001a114423381a80400535a3ea9a--