ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMBARI-21239) Ambari leaves a lot of processes from Hive alert on the machine
Date Tue, 13 Jun 2017 13:58:00 GMT
Andrew Onischuk created AMBARI-21239:
----------------------------------------

             Summary: Ambari leaves a lot of processes from Hive alert on the machine
                 Key: AMBARI-21239
                 URL: https://issues.apache.org/jira/browse/AMBARI-21239
             Project: Ambari
          Issue Type: Bug
            Reporter: Andrew Onischuk
            Assignee: Andrew Onischuk
             Fix For: 2.5.2


I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
the database VM storing the metadata after the cluster had been runing for
almost 2 weeks without issues. What I observed is that the instance in a few
hours got really slow. Upon investigation found hundreds of Beeline processes
like this:

    
    
    ambari-+ 3439770 3439769  0 06:19 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
 -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
    ambari-+ 3439814 3439770  0 06:19 ?        00:00:42 /usr/lib/jvm/java-openjdk/bin/java
-Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop
-Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
-e
    ambari-+ 3439816 3439770  0 06:19 ?        00:00:00 awk {print}
    ambari-+ 3439817 3439770  0 06:19 ?        00:00:00 grep -i -e Connection refused -e Invalid
URL
    ambari-+ 3440645 3440644  0 06:22 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
 -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
    ambari-+ 3440685 3440645  0 06:22 ?        00:00:42 /usr/lib/jvm/java-openjdk/bin/java
-Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop
-Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
-e
    ambari-+ 3440686 3440645  0 06:22 ?        00:00:00 awk {print}
    ambari-+ 3440687 3440645  0 06:22 ?        00:00:00 grep -i -e Connection refused -e Invalid
URL
    ambari-+ 3441185 3441184  0 06:25 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
 -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
    ambari-+ 3441224 3441185  0 06:25 ?        00:00:46 /usr/lib/jvm/java-openjdk/bin/java
-Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop
-Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
-e
    

Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in
them and owned by ambari found 309 processes.  
The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't
accessible as the DB host serving the metadata went down.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message