ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitry Lysnichenko (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-12374) Unable to Start NameNode in HA Mode On HDP 2.0
Date Tue, 21 Jul 2015 17:05:05 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dmitry Lysnichenko updated AMBARI-12374:
----------------------------------------
    Attachment: AMBARI-12374.patch

> Unable to Start NameNode in HA Mode On HDP 2.0
> ----------------------------------------------
>
>                 Key: AMBARI-12374
>                 URL: https://issues.apache.org/jira/browse/AMBARI-12374
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>            Reporter: Dmitry Lysnichenko
>            Assignee: Dmitry Lysnichenko
>             Fix For: 2.1.1
>
>         Attachments: AMBARI-12374.patch
>
>
> When starting an HA NameNode cluster on HDP 2.0, the following error is seen:
> {code}
> 2015-07-07 16:02:56,371 - Getting jmx metrics from NN failed. URL: http://c6401.ambari.apache.org:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py",
line 41, in get_value_from_jmx
>     return data_dict["beans"][0][property]
> IndexError: list index out of range
> 2015-07-07 16:02:56,396 - Getting jmx metrics from NN failed. URL: http://c6402.ambari.apache.org:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py",
line 41, in get_value_from_jmx
>     return data_dict["beans"][0][property]
> IndexError: list index out of range
> Traceback (most recent call last):
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py",
line 316, in <module>
>     NameNode().execute()
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 216, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py",
line 81, in start
>     namenode(action="start", rolling_restart=rolling_restart, env=env)
>   File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89,
in thunk
>     return fn(*args, **kwargs)
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py",
line 141, in namenode
>     create_hdfs_directories(is_active_namenode_cmd)
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py",
line 198, in create_hdfs_directories
>     only_if=check
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157,
in __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
line 390, in action_create_on_execute
>     self.action_delayed("create")
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
line 387, in action_delayed
>     self.get_hdfs_resource_executor().action_delayed(action_name, self)
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
line 239, in action_delayed
>     main_resource.resource.security_enabled, main_resource.resource.logoutput)
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
line 126, in __init__
>     security_enabled, run_user)
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/namenode_ha_utils.py",
line 113, in get_property_for_active_namenode
>     raise Fail("There is no active namenodes.")
> resource_management.core.exceptions.Fail: There is no active namenodes.
> {code}
> Although the NameNode does actually start, a failure is recorded in the request, stopping
the rest of the cluster from coming up. This probably because the JMX properties for Active
and Standby NameNode are different in HDP 2.0 vs HDP 2.1+:
> {code:title=active jmx}
> {
>     "name" : "Hadoop:service=NameNode,name=FSNamesystem",
>     "modelerType" : "FSNamesystem",
>     "tag.Context" : "dfs",
>     "tag.HAState" : "active",
> {code}
> {code:title=standby jmx}
> {
>     "name" : "Hadoop:service=NameNode,name=FSNamesystem",
>     "modelerType" : "FSNamesystem",
>     "tag.Context" : "dfs",
>     "tag.HAState" : "standby",
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message