ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandor Magyari (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-14409) Blueprints: Kerberos deployments fail intermittently due to invalid keytabs
Date Thu, 17 Dec 2015 08:06:46 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sandor Magyari updated AMBARI-14409:
------------------------------------
    Attachment: AMBARI-14409.patch

> Blueprints: Kerberos deployments fail intermittently due to invalid keytabs
> ---------------------------------------------------------------------------
>
>                 Key: AMBARI-14409
>                 URL: https://issues.apache.org/jira/browse/AMBARI-14409
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 2.2.0
>            Reporter: Sandor Magyari
>            Assignee: Sandor Magyari
>            Priority: Critical
>             Fix For: 2.2.1
>
>         Attachments: AMBARI-14409.patch
>
>
> The basic problem appears to be a concurrency problem, in which keytabs are sometimes
generated incorrectly.  This results in certain keytabs across the cluster having different
created/modified times in the keytabs, which can cause Kerberos Preauthentication failures,
such as the following:
> {code}
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab
ambari-qa-clusterone@EXAMPLE.COM;' returned 1. kinit: Password incorrect while getting initial
credentials
> Traceback (most recent call last):
>   File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py",
line 179, in <module>
>     ZookeeperServer().execute()
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 217, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py",
line 55, in start
>     zookeeper_service(action='start', upgrade_type=upgrade_type)
>   File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89,
in thunk
>     return fn(*args, **kwargs)
>   File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py",
line 54, in zookeeper_service
>     user=params.smokeuser
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154,
in __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
158, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
121, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 238, in action_run
>     tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70,
in inner
>     result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92,
in checked_call
>     tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140,
in _call_wrapper
>     result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291,
in _call
>     raise Fail(err_msg)
> {code}
> The basic underlying problem appears to be invalid keytabs.  Looking to the keytabs on
the failing node:
> {code}
> [root@rwntest-2 keytabs]# klist -kt smokeuser.headless.keytab
> Keytab name: FILE:smokeuser.headless.keytab
> KVNO Timestamp         Principal
> ---- ----------------- --------------------------------------------------------
>    1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
>    1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
>    1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
>    1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
>    1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
> {code}
> whereas other nodes in the cluster report the following value for the same keytab:
> {code}
> [root@rwntest-4 keytabs]# klist -kt smokeuser.headless.keytab
> Keytab name: FILE:smokeuser.headless.keytab
> KVNO Timestamp         Principal
> ---- ----------------- --------------------------------------------------------
>    4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
>    4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
>    4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
>    4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
>    4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
> {code}
> It looks like these keytabs should have the same timestamps, and this is causing the
Kerberos failures at cluster start up. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message