Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of stevel@hortonworks.com
 designates 209.85.128.182 as permitted sender)
MIME-Version: 1.0
Date: Wed, 29 Jan 2014 16:33:53 +0000
Message-ID: 
 <CA+4kjVuviKSvJmuc3xENkYEhYqqU6FJr9WrV1wtwZdxiCz-vkg@mail.gmail.com>
Subject: regression in 2.4? YARN severs on secure cluster startup
From: Steve Loughran <stevel@hortonworks.com>
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a1136b264ad8cb504f11e8176

--001a1136b264ad8cb504f11e8176
Content-Type: text/plain; charset=UTF-8

I'm just switching over to use the 2.4-SNAPSHOT in a secured pseudo-dist
cluster, and now the services are failing to come up because the web
principals haven't been defined. Example


2014-01-29 15:42:58,558 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter (class=org.apache.hadoop.ht
tp.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter (class=org.apache.hadoop.ht
tp.lib.StaticUserWebFilter$StaticUserFilter) to context static
2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter (class=org.apache.hadoop.ht
tp.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2014-01-29 15:42:58,630 ERROR org.apache.hadoop.http.HttpServer2: *WebHDFS
and security are enabled, but configuration proper*
*ty 'dfs.web.authentication.kerberos.principal' is not set.*
2014-01-29 15:42:58,630 INFO org.apache.hadoop.http.HttpServer2: Added
filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.Aut
hFilter)
2014-01-29 15:42:58,631 INFO org.apache.hadoop.http.HttpServer2:
addJerseyResourcePackage: packageName=org.apache.hadoop.hdf
s.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
2014-01-29 15:42:58,658 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to getDelegationToken
2014-01-29 15:42:58,662 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to renewDelegationToken
2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to cancelDelegationToken
2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to fsck
2014-01-29 15:42:58,671 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to getimage
2014-01-29 15:42:58,748 INFO org.apache.hadoop.http.HttpServer2: Jetty
bound to port 50070
2014-01-29 15:42:58,748 INFO org.mortbay.log: jetty-6.1.26
2014-01-29 15:42:58,941 INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Login using key
tab /home/stevel/conf/hdfs.keytab, for principal HTTP/ubuntu@COTHAM
2014-01-29 15:42:58,981 INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Initialized, pr
incipal [HTTP/ubuntu@COTHAM] from keytab [/home/stevel/conf/hdfs.keytab]
2014-01-29 15:42:58,981 WARN
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
'signature.secret' confi
guration not set, using a random value as secret
2014-01-29 15:42:58,982 WARN org.mortbay.log: failed SPNEGO:
javax.servlet.ServletException: javax.servlet.ServletException:
 Principal not defined in configuration
2014-01-29 15:42:58,982 WARN org.mortbay.log: Failed startup of context
org.mortbay.jetty.webapp.WebAppContext@167a465{/,fil
e:/home/stevel/hadoop/share/hadoop/hdfs/webapps/hdfs}
javax.servlet.ServletException: javax.servlet.ServletException: Principal
not defined in configuration
        at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler
.java:203)
:

YARN is the same but without the text telling me what config option I have
to set (i.e no equivalent of https://issues.apache.org/jira/browse/HDFS-3813)

-29 16:04:33,908 INFO
org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor
thread interrupted
2014-01-29 16:04:33,908 INFO
org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
thread interrupted
2014-01-29 16:04:33,908 ERROR
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
InterruptedExcpetion recieved for ExpiredTokenRemover thread
java.lang.InterruptedException: sleep interrupted
2014-01-29 16:04:33,909 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
to standby state
2014-01-29 16:04:33,909 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
starting ResourceManager
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:250)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:775)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:866)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:995)
Caused by: java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:245)
... 4 more
Caused by: javax.servlet.ServletException: javax.servlet.ServletException:
Principal not defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:787)
... 5 more
Caused by: javax.servlet.ServletException: Principal not defined in
configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:164)
... 21 more

This cluster config (
https://github.com/hortonworks/hoya/tree/master/hoya-funtest/src/test/configs/ubuntu-secure)
did work on Hadoop-2.2, which implies that there is some change in
Hadoop
2.4 that is incompatible to the extent that you now need to declare some
principals you didn't need to set now need to be


1. it Looks like HDFS-3604, enable webhdfs is the issue that stops the NN
and DN  -IMO it should be marked as an incompatible change, as it stops
clusters that used to start, from starting.
2. what's causing this in YARN?

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

--001a1136b264ad8cb504f11e8176--