ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Sekhon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-8785) Resource Manager HA Kerberos principal not handled and user not notified of requirement
Date Thu, 18 Dec 2014 12:19:13 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hari Sekhon updated AMBARI-8785:
--------------------------------
    Description: 
When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari fails to tell
the user about the required Kerberos principal + keytab for the new Resource Manager instance.

As as result the new Resource Manager fails to start with the following logs:
{code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1043))
- Error starting ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab /etc/security/keytabs/rm.service.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
        ... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
        ... 5 more
2014-12-18 11:39:06,383 INFO  resourcemanager.ResourceManager (StringUtils.java:run(640))
- SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
************************************************************/
{code}
The fix is quite simple for experienced cluster administrators, create the principal, export
the keytab, mimic the original RM permissions of yarn:hadoop 620 (not really correct octal
but that's a separate issue I've already raised to fix keytab perms AMBARI-8138) and then
it will start up.

This is sort of related to AMBARI-8610 where Ambari should export a CSV for new hosts/services
to allow existing automation scripts to generate the principals and and distributed the keytabs
such as the basic kerberos_setup.sh shipped with Ambari or the more real world FreeIPA one
I publish on my github (https://github.com/harisekhon/toolbox).

Except in this instance it's not a new service, it's just HA enablement where it doesn't notify
users that RM HA will require a new Kerberos principal + keytab to allow them to do that,
never mind provide a subset CSV for passing to a kerberos setup script. This issue will also
affect NN HA if Ambari ever allows setting up NN HA without disabling Kerberos first ().

Regards,

Hari Sekhon
http://www.linkedin.com/in/harisekhon

  was:
When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari fails to tell
the user about the required Kerberos principal + keytab for the new Resource Manager instance.

As as result the new Resource Manager fails to start with the following logs:
{code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1043))
- Error starting ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab /etc/security/keytabs/rm.service.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
        ... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
        ... 5 more
2014-12-18 11:39:06,383 INFO  resourcemanager.ResourceManager (StringUtils.java:run(640))
- SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
************************************************************/
{code}
The fix is quite simple for experienced cluster administrators, create the principal, export
the keytab, mimic the original RM permissions of yarn:hadoop 620 (not really correct octal
but that's a separate issue I've already raised to fix keytab perms AMBARI-8138) and then
it will start up.

This is sort of related to AMBARI-8610 where Ambari should export a CSV for new hosts/services
to allow existing automation scripts to generate the principals and and distributed the keytabs
such as the basic kerberos_setup.sh shipped with Ambari or the more real world FreeIPA one
I publish on my github (https://github.com/harisekhon/toolbox).

Except in this instance it's not a new service, it's just HA enablement where it doesn't notify
users that RM HA will require a new Kerberos principal + keytab to allow them to do that,
never mind provide a subset CSV for passing to a kerberos setup script. This issue will also
affect NN HA if Ambari ever allows setting up NN HA without disabling Kerberos first.

Regards,

Hari Sekhon
http://www.linkedin.com/in/harisekhon


> Resource Manager HA Kerberos principal not handled and user not notified of requirement
> ---------------------------------------------------------------------------------------
>
>                 Key: AMBARI-8785
>                 URL: https://issues.apache.org/jira/browse/AMBARI-8785
>             Project: Ambari
>          Issue Type: Bug
>    Affects Versions: 1.7.0
>         Environment: HDP 2.1
>            Reporter: Hari Sekhon
>
> When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari fails to
tell the user about the required Kerberos principal + keytab for the new Resource Manager
instance.
> As as result the new Resource Manager fails to start with the following logs:
> {code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1043))
- Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
>         at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
> Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab /etc/security/keytabs/rm.service.keytab
>         at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
>         at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
>         ... 2 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
>         at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
>         at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
>         at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
>         at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
>         at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
>         at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
>         at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
>         at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
>         ... 5 more
> 2014-12-18 11:39:06,383 INFO  resourcemanager.ResourceManager (StringUtils.java:run(640))
- SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
> ************************************************************/
> {code}
> The fix is quite simple for experienced cluster administrators, create the principal,
export the keytab, mimic the original RM permissions of yarn:hadoop 620 (not really correct
octal but that's a separate issue I've already raised to fix keytab perms AMBARI-8138) and
then it will start up.
> This is sort of related to AMBARI-8610 where Ambari should export a CSV for new hosts/services
to allow existing automation scripts to generate the principals and and distributed the keytabs
such as the basic kerberos_setup.sh shipped with Ambari or the more real world FreeIPA one
I publish on my github (https://github.com/harisekhon/toolbox).
> Except in this instance it's not a new service, it's just HA enablement where it doesn't
notify users that RM HA will require a new Kerberos principal + keytab to allow them to do
that, never mind provide a subset CSV for passing to a kerberos setup script. This issue will
also affect NN HA if Ambari ever allows setting up NN HA without disabling Kerberos first
().
> Regards,
> Hari Sekhon
> http://www.linkedin.com/in/harisekhon



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message