flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: JVM Non Heap Memory
Date Tue, 29 Nov 2016 16:57:08 GMT
Hey Daniel!

Thanks for reporting this. Unbounded growth of non-heap memory is not expected.  What kind
of Threads are you seeing being spawned/lingering around?

As a first step, could you try to disable checkpointing and see how it behaves afterwards?

– Ufuk

On 29 November 2016 at 17:32:32, Daniel Santos (dsantos@cryptolab.net) wrote:
> Hello,
>  
> Nope I am using Hadoop HDFS, as state backend, Kafka, as source, and a
> HttpClient as a Sink, also Kafka as Sink.
> So it's possible that the state backend is the culprit?
>  
> Curious thing is even when no jobs are running streaming or otherwise,
> the JVM Non-HEAP stays the same.
> Which I find it odd.
>  
> Another curious thing is that it's proportional to an increase of JVM
> thread's number.
> Whenever there are more JVM threads running there is also more JVM
> Non-HEAP being used, which makes sense.
> But threads stick around never decreasing, too, likewise JVM Non-HEAP
> memory.
>  
> These observations described are based on what flink's metrics are being
> sent and recorded to our graphite's system.
>  
> Best Regards,
>  
> Daniel Santos
>  
> On 11/29/2016 04:04 PM, Cliff Resnick wrote:
> > Are you using the RocksDB backend in native mode? If so then the
> > off-heap memory may be there.
> >
> > On Tue, Nov 29, 2016 at 9:54 AM, > > > wrote:
> >
> > i have the same problem,but i put the flink job into yarn.
> > but i put the job into yarn on the computer 22,and the job can
> > success run,and the jobmanager is 79 and taskmanager is 69,they
> > three different compu345ter,
> > however,on computer 22,the pid=3463,which is the job that put into
> > yarn,is have 2.3g memory,15% of total,
> > the commend is : ./flink run -m yarn-cluster -yn 1 -ys 1 -yjm 1024
> > -ytm 1024 ....
> > why in conputer 22,has occupy so much momory?the job is running
> > computer 79 and computer 69.
> > What would be the possible causes of such behavior ?
> > Best Regards,
> > ----- 原始邮件 -----
> > 发件人:Daniel Santos > > >
> > 收件人:user@flink.apache.org  
> > 主题:JVM Non Heap Memory
> > 日期:2016年11月29日 22点26分
> >
> >
> > Hello,
> > Is it common to have high usage of Non-Heap in JVM ?
> > I am running flink in stand-alone cluster and in docker, with each
> > docker bieng capped at 6G of memory.
> > I have been struggling to keep memory usage in check.
> > The non-heap increases to no end. It start with just 100MB of
> > usage and
> > after a day it reaches to 1,3GB.
> > Then evetually reaches to 2GB and then eventually the docker is
> > killed
> > because it has reached the memory limit.
> > My configuration for each flink task manager is the following :
> > ----------- flink-conf.yaml --------------
> > taskmanager.heap.mb: 3072
> > taskmanager.numberOfTaskSlots: 8
> > taskmanager.memory.preallocate: false
> > taskmanager.network.numberOfBuffers: 12500
> > taskmanager.memory.off-heap: false
> > ---------------------------------------------
> > What would be the possible causes of such behavior ?
> > Best Regards,
> > Daniel Santos
> >
> >
>  
>  


Mime
View raw message