ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumit Mohanty (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-15506) Support for multiple Namservices in namenode_ha_utils.py
Date Sat, 09 Apr 2016 03:33:25 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233331#comment-15233331
] 

Sumit Mohanty commented on AMBARI-15506:
----------------------------------------

[~hkropp] The patch looks good to me. I run into two UT failures when I apply the patch -
I think you can add a check such as 

{code}
name_service = None
if 'dfs.internal.nameservices' in hdfs_site:
  name_service = hdfs_site['dfs.internal.nameservices']
{code}

UT failures
{code}
ERROR: test_start_default (test_oozie_server.TestOozieServer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/test/python/mock/mock.py",
line 1199, in patched
    return func(*args, **keywargs)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py",
line 519, in test_start_default
    call_mocks = call_mocks
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/test/python/stacks/utils/RMFTestCase.py",
line 145, in executeScript
    method(RMFTestCase.env, *command_args)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/test/python/stacks/utils/../../../../main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py",
line 96, in start
    oozie_service(action='start', upgrade_type=upgrade_type)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/ambari_commons/os_family_impl.py",
line 89, in thunk
    return fn(*args, **kwargs)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py",
line 106, in oozie_service
    util = WebHDFSUtil(params.hdfs_site, params.oozie_user, params.security_enabled)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py",
line 133, in __init__
    security_enabled, run_user)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py",
line 153, in get_property_for_active_namenode
    if is_ha_enabled(hdfs_site):
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py",
line 120, in is_ha_enabled
    dfs_ha_nameservices = get_nameservice(hdfs_site)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py",
line 187, in get_nameservice
    if not name_service:
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py",
line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations
dictionary!")
Fail: Configuration parameter 'dfs.internal.nameservices' was not found in configurations
dictionary!

ERROR: test_start_secured (test_oozie_server.TestOozieServer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/test/python/mock/mock.py",
line 1199, in patched
    return func(*args, **keywargs)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py",
line 614, in test_start_secured
    call_mocks = call_mocks
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/test/python/stacks/utils/RMFTestCase.py",
line 145, in executeScript
    method(RMFTestCase.env, *command_args)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/test/python/stacks/utils/../../../../main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py",
line 96, in start
    oozie_service(action='start', upgrade_type=upgrade_type)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/ambari_commons/os_family_impl.py",
line 89, in thunk
    return fn(*args, **kwargs)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py",
line 106, in oozie_service
    util = WebHDFSUtil(params.hdfs_site, params.oozie_user, params.security_enabled)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py",
line 133, in __init__
    security_enabled, run_user)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py",
line 153, in get_property_for_active_namenode
    if is_ha_enabled(hdfs_site):
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py",
line 120, in is_ha_enabled
    dfs_ha_nameservices = get_nameservice(hdfs_site)
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py",
line 187, in get_nameservice
    if not name_service:
  File "/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py",
line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations
dictionary!")
Fail: Configuration parameter 'dfs.internal.nameservices' was not found in configurations
dictionary!

----------------------------------------------------------------------
{code}

Easy way to run UTs for stack changes - modify [ROOT] to your local path:
{code}
cd [ROOT]/ambari-server/src/test/python

export PYTHONPATH=/Users/smohanty/enlistments/asf-ambari/ambari-common/src/main/python:/Users/smohanty/enlistments/asf-ambari/ambari-agent/src/main/python:[ROOT]/ambari-common/src/main/python/ambari_jinja2:[ROOT]/ambari-common/src/main/python/ambari_commons:[ROOT]/ambari-common/src/test/python:[ROOT]/ambari-server/src/main/python:[ROOT]/ambari-server/src/main/python/ambari-server-state:[ROOT]/ambari-server/src/main/resources/custom_actions:[ROOT]/ambari-server/src/main/resources/scripts:[ROOT]/ambari-server/src/test/python

[ROOT]/ambari-common/src/main/unix/ambari-python-wrap unitTests.py
{code} 

> Support for multiple Namservices in namenode_ha_utils.py
> --------------------------------------------------------
>
>                 Key: AMBARI-15506
>                 URL: https://issues.apache.org/jira/browse/AMBARI-15506
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-agent
>    Affects Versions: trunk
>         Environment: HDFS HA
>            Reporter: Henning Kropp
>             Fix For: 2.4.0
>
>         Attachments: AMBARI-15506, AMBARI-15506_1, namenode_ha_utils.py
>
>
> Currently Ambari does not support the definition of multiple Nameservices. It is always
assumed {{hdfs_site\['dfs.nameservices'\]}} is just a string defining one nameservice.
> Multiple nameservices can be configured for exmaple to support seamless distcp between
two HA clusters. The nameservices are defined as a comma separated list in {{hdfs_site\['dfs.nameservices'\]}}.

> This patch introduces the method {{get_nameservice(hdfs_site)}}, which splits the value
into in an array and identifies the nameservice for the current cluster with what is set in
{{hdfs_site\['dfs.namenode.shared.edits.dir'\]}}. By default the first nameservice is returned
or empty.
> To verify the current namservice it would be preferred to use {{fs.defaultFS}} in {{core-site}},
but getting this config into {{namenode_ha_utils.py}} seems more involved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message