hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (Naga)" <garlanaganarasi...@huawei.com>
Subject RE: YARN timelineserver process taking 600% CPU
Date Wed, 30 Sep 2015 14:23:04 GMT
Hi ,

Whats the size of Store Files?
Since when is it running ? how many applications have been run since it has been started ?
Whats the value of "yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms" ?

+ Naga
________________________________
From: Krzysztof Zarzycki [k.zarzycki@gmail.com]
Sent: Wednesday, September 30, 2015 19:20
To: user@hadoop.apache.org
Subject: YARN timelineserver process taking 600% CPU

Hi there Hadoopers,
I have a serious issue with my installation of Hadoop & YARN in version 2.7.1 (HDP 2.3).
The timelineserver process ( more precisely org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
class) takes over 600% of CPU, generating enormous load on my master node. I can't guess why
it happens.

First, I run the timelineserver using java 8, thought that this was an issue. But no, I started
timelineserver now with use of java 7 and still the problem is the same.

My cluster is tiny- it consists of:
- 2 HDFS nodes
- 2 HBase RegionServers
- 2 Kafkas
- 2 Spark nodes
- 8 Spark Streaming jobs, processing around 100 messages/second TOTAL.

I'll be very grateful for your help here. If you need any more info, please write.
I also attach yarn-site.xml grepped to options related to timeline server.

And here is a command of timeline that I see from ps :
/usr/java/jdk1.7.0_79/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.3.0.0-2557
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn -Dhadoop.log.file=yarn-yarn-timelineserver-hd-master-a01.log
-Dyarn.log.file=yarn-yarn-timelineserver-hd-master-a01.log -Dyarn.home.dir= -Dyarn.id.str=yarn
-Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA -Djava.library.path=:/usr/hdp/2.3.0.0-2557/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2557/hadoop/lib/native:/usr/hdp/2.3.0.0-2557/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2557/hadoop/lib/native
-Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn
-Dhadoop.log.file=yarn-yarn-timelineserver-hd-master-a01.log -Dyarn.log.file=yarn-yarn-timelineserver-hd-master-a01.log
-Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-timelineserver -Dhadoop.home.dir=/usr/hdp/2.3.0.0-2557/hadoop
-Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA -Djava.library.path=:/usr/hdp/2.3.0.0-2557/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2557/hadoop/lib/native:/usr/hdp/2.3.0.0-2557/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2557/hadoop/lib/native
-classpath /usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2.3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/.//*:::/usr/share/java/mysql-connector-java.jar::/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-yarn-timelineserver/.//*:/usr/hdp/current/hadoop-yarn-timelineserver/lib/*:/usr/hdp/current/hadoop-client/conf/timelineserver-config/log4j.properties
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer


Thanks!
Krzysztof


Mime
View raw message