flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <rimin...@sina.cn>
Subject flink-job-in-yarn,has max memory
Date Tue, 29 Nov 2016 11:43:27 GMT
Hi,     i have a flink job,and abt assembly to get a jar file,so i put it to yarn and run it,use
the follow commend:------------------------------------------------------------------------/home/www/flink-1.1.1/bin/flink
run \-m yarn-cluster \-yn 1 \-ys 2 \-yjm 4096 \-ytm 4096 \--class skRecomm.SkProRecommFlink
\--classpath file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar \--classpath file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
\--classpath file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar \--classpath file:///opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar
\--classpath file:///opt/cloudera/parcels/CDH/lib/hbase/lib/guava-12.0.1.jar \/home/www/flink-mining/deploy/zx_article-7cffb87.jar
-----------------------------------------------------------------------------------
the commend is in  supervisor on a computer(*,*,*,22),----------------------------and  flink/conf/flink-conf.yaml,i
set those pargam,------------------------------------------fs.hdfs.hadoopconf: /etc/hadoop/conf/jobmanager.web.port:
8081parallelism.default: 1taskmanager.memory.preallocate: falsetaskmanager.numberOfTaskSlots:
1taskmanager.heap.mb: 512jobmanager.heap.mb: 256arallelism.default: 1jobmanager.rpc.port:
6123jobmanager.rpc.address: localhost

------------------------------------------the job is success, can find follow message in yarn
monitor,
flink.base.dir.path /data1/yarn/nm/usercache/work/appcache/application_1472623395420_36719/container_e03_1472623395420_36719_01_000001fs.hdfs.hadoopconf
/etc/hadoop/conf/jobmanager.heap.mb	256jobmanager.rpc.address	*.*.*.79  -----(is not *.*.*.22,and
taskmanager is *.*.*.69)jobmanager.rpc.port	32987jobmanager.web.port	0parallelism.default
       1recovery.zookeeper.path.namespace	application_1472623395420_36719taskmanager.heap.mb
512taskmanager.memory.preallocate	falsetaskmanager.numberOfTaskSlots	1
-----------------------------------------------------OverviewData Port	All Slots	Free Slots
CPU Cores	Physical Memory	Free Memory	Flink Managed Memory30471	            2	           0
            32	         189 GB	        2.88 GB	            1.96 GB-----------------------------------------------------------------------------------------------------------------------MemoryJVM
(Heap/Non-Heap)Type	Committed	        Initial	   MaximumHeap	 2.92 GB	             3.00 GB
    2.92 GBNon-Heap  53.4 MB	             23.4 MB	    130 MBTotal	  2.97 GB	             3.02
GB	    3.04 GB-----------------------------------------------------------------Outside JVMType
Count	Used	CapacityDirect	510	       860 KB	860 KBMapped	0	         0 B	         0 B-------------------------------------------------------------------
i find in computer(*,*,*,22),the pid=345 has 2.36g memory,and the pid=345 is the job that
 from supervisor run,
i really do not know why ?the job was run in yarn ,why occupy so much memory in computer(*.*.*.22),i
just run the job in computer(*.*.*.22).
thank you answer my question.
Mime
View raw message