ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-21239) Ambari leaves a lot of processes from Hive alert on the machine
Date Tue, 13 Jun 2017 14:51:00 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047980#comment-16047980
] 

Hudson commented on AMBARI-21239:
---------------------------------

FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #7622 (See [https://builds.apache.org/job/Ambari-trunk-Commit/7622/])
AMBARI-21239. Ambari leaves a lot of processes from Hive alert on the (aonishuk: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=31ac3c21b71868df4f4e2ac82e72defa79767bd6])
* (edit) ambari-common/src/main/python/resource_management/libraries/functions/hive_check.py
* (edit) ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_service_check.py


> Ambari leaves a lot of processes from Hive alert on the machine
> ---------------------------------------------------------------
>
>                 Key: AMBARI-21239
>                 URL: https://issues.apache.org/jira/browse/AMBARI-21239
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Andrew Onischuk
>            Assignee: Andrew Onischuk
>             Fix For: 2.5.2
>
>         Attachments: AMBARI-21239.patch
>
>
> I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
> the database VM storing the metadata after the cluster had been runing for
> almost 2 weeks without issues. What I observed is that the instance in a few
> hours got really slow. Upon investigation found hundreds of Beeline processes
> like this:
>     
>     
>     ambari-+ 3439770 3439769  0 06:19 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
 -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>     ambari-+ 3439814 3439770  0 06:19 ?        00:00:42 /usr/lib/jvm/java-openjdk/bin/java
-Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop
-Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
-e
>     ambari-+ 3439816 3439770  0 06:19 ?        00:00:00 awk {print}
>     ambari-+ 3439817 3439770  0 06:19 ?        00:00:00 grep -i -e Connection refused
-e Invalid URL
>     ambari-+ 3440645 3440644  0 06:22 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
 -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>     ambari-+ 3440685 3440645  0 06:22 ?        00:00:42 /usr/lib/jvm/java-openjdk/bin/java
-Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop
-Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
-e
>     ambari-+ 3440686 3440645  0 06:22 ?        00:00:00 awk {print}
>     ambari-+ 3440687 3440645  0 06:22 ?        00:00:00 grep -i -e Connection refused
-e Invalid URL
>     ambari-+ 3441185 3441184  0 06:25 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
 -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>     ambari-+ 3441224 3441185  0 06:25 ?        00:00:46 /usr/lib/jvm/java-openjdk/bin/java
-Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop
-Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
-e
>     
> Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in
> them and owned by ambari found 309 processes.  
> The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't
> accessible as the DB host serving the metadata went down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message