ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Howard (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-21567) Ambari reports GREEN zookeeper when out of disk space prevents quorum
Date Mon, 24 Jul 2017 20:26:00 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-21567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Howard updated AMBARI-21567:
------------------------------------
    Description: 
* Ambari reports Zookeeper status as GREEN, even though ZK is not happy and has not reached
a quorum. 
* Ambari also shows YARN ResourceManager as GREEN, even though it doesn't fully start ...
because it is waiting for Zookeeper
* YARN NodeManagers will try to start ... Ambari will report as GREEN and started for a few
seconds, but then shows them as stopped. 
* YARN NodeManagers are failing to start because ResourceManager is not listening on port
8141 ... because Zookeeper is still waiting for quorum. 
* root cause is that one of the  Zookeeper nodes ran out of disk space on /opt/hadoop/zookeeper
volume ... which prevented a quorum from being formed

Seems like Ambari dashboard should catch the fact that Zookeeper was not in a good state.


In our case, it would have saved significant time in diagnosing the problem. 


  was:
* Ambari reports Zookeeper status as GREEN, even though ZK is not happy and has not reached
a quorum. 
* Ambari also shows YARN ResourceManager as GREEN, even though it doesn't start ... because
it is waiting for Zookeeper
* YARN NodeManagers will try to start ... Ambari will report as GREEN and started for a few
seconds, but then shows them as stopped. 
* root cause is that one of the  Zookeeper nodes ran out of disk space on /opt/hadoop/zookeeper
volume ... which prevented a quorum from being formed

Seems like Ambari dashboard should catch the fact that Zookeeper was not in a good state.


In our case, it would have saved significant time in diagnosing the problem. 



> Ambari reports GREEN zookeeper when out of disk space prevents quorum
> ---------------------------------------------------------------------
>
>                 Key: AMBARI-21567
>                 URL: https://issues.apache.org/jira/browse/AMBARI-21567
>             Project: Ambari
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>         Environment: Ambari 2.2.0
> IBM Open Platform 4.2.0.0
> CentOS 6.8
>            Reporter: Michael Howard
>
> * Ambari reports Zookeeper status as GREEN, even though ZK is not happy and has not reached
a quorum. 
> * Ambari also shows YARN ResourceManager as GREEN, even though it doesn't fully start
... because it is waiting for Zookeeper
> * YARN NodeManagers will try to start ... Ambari will report as GREEN and started for
a few seconds, but then shows them as stopped. 
> * YARN NodeManagers are failing to start because ResourceManager is not listening on
port 8141 ... because Zookeeper is still waiting for quorum. 
> * root cause is that one of the  Zookeeper nodes ran out of disk space on /opt/hadoop/zookeeper
volume ... which prevented a quorum from being formed
> Seems like Ambari dashboard should catch the fact that Zookeeper was not in a good state.

> In our case, it would have saved significant time in diagnosing the problem. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message