hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Lowe <jl...@yahoo-inc.com>
Subject Re: regression in 2.4? YARN severs on secure cluster startup
Date Wed, 29 Jan 2014 17:13:51 GMT
The RM issue should be fixed by YARN-1600 which I just committed this 
morning.

Jason

On 01/29/2014 10:33 AM, Steve Loughran wrote:
> I'm just switching over to use the 2.4-SNAPSHOT in a secured pseudo-dist
> cluster, and now the services are failing to come up because the web
> principals haven't been defined. Example
>
>
> 2014-01-29 15:42:58,558 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter (class=org.apache.hadoop.ht
> tp.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
> 2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter (class=org.apache.hadoop.ht
> tp.lib.StaticUserWebFilter$StaticUserFilter) to context static
> 2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter (class=org.apache.hadoop.ht
> tp.lib.StaticUserWebFilter$StaticUserFilter) to context logs
> 2014-01-29 15:42:58,630 ERROR org.apache.hadoop.http.HttpServer2: *WebHDFS
> and security are enabled, but configuration proper*
> *ty 'dfs.web.authentication.kerberos.principal' is not set.*
> 2014-01-29 15:42:58,630 INFO org.apache.hadoop.http.HttpServer2: Added
> filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.Aut
> hFilter)
> 2014-01-29 15:42:58,631 INFO org.apache.hadoop.http.HttpServer2:
> addJerseyResourcePackage: packageName=org.apache.hadoop.hdf
> s.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
> pathSpec=/webhdfs/v1/*
> 2014-01-29 15:42:58,658 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to getDelegationToken
> 2014-01-29 15:42:58,662 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to renewDelegationToken
> 2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to cancelDelegationToken
> 2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to fsck
> 2014-01-29 15:42:58,671 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to getimage
> 2014-01-29 15:42:58,748 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 50070
> 2014-01-29 15:42:58,748 INFO org.mortbay.log: jetty-6.1.26
> 2014-01-29 15:42:58,941 INFO
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
> Login using key
> tab /home/stevel/conf/hdfs.keytab, for principal HTTP/ubuntu@COTHAM
> 2014-01-29 15:42:58,981 INFO
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
> Initialized, pr
> incipal [HTTP/ubuntu@COTHAM] from keytab [/home/stevel/conf/hdfs.keytab]
> 2014-01-29 15:42:58,981 WARN
> org.apache.hadoop.security.authentication.server.AuthenticationFilter:
> 'signature.secret' confi
> guration not set, using a random value as secret
> 2014-01-29 15:42:58,982 WARN org.mortbay.log: failed SPNEGO:
> javax.servlet.ServletException: javax.servlet.ServletException:
>   Principal not defined in configuration
> 2014-01-29 15:42:58,982 WARN org.mortbay.log: Failed startup of context
> org.mortbay.jetty.webapp.WebAppContext@167a465{/,fil
> e:/home/stevel/hadoop/share/hadoop/hdfs/webapps/hdfs}
> javax.servlet.ServletException: javax.servlet.ServletException: Principal
> not defined in configuration
>          at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler
> .java:203)
> :
>
> YARN is the same but without the text telling me what config option I have
> to set (i.e no equivalent of https://issues.apache.org/jira/browse/HDFS-3813)
>
> -29 16:04:33,908 INFO
> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor
> thread interrupted
> 2014-01-29 16:04:33,908 INFO
> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
> thread interrupted
> 2014-01-29 16:04:33,908 ERROR
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> InterruptedExcpetion recieved for ExpiredTokenRemover thread
> java.lang.InterruptedException: sleep interrupted
> 2014-01-29 16:04:33,909 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2014-01-29 16:04:33,909 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> starting ResourceManager
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:250)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:775)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:866)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:995)
> Caused by: java.io.IOException: Unable to initialize WebAppContext
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:245)
> ... 4 more
> Caused by: javax.servlet.ServletException: javax.servlet.ServletException:
> Principal not defined in configuration
> at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
> at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
> at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:224)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:787)
> ... 5 more
> Caused by: javax.servlet.ServletException: Principal not defined in
> configuration
> at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:164)
> ... 21 more
>
> This cluster config (
> https://github.com/hortonworks/hoya/tree/master/hoya-funtest/src/test/configs/ubuntu-secure)
> did work on Hadoop-2.2, which implies that there is some change in
> Hadoop
> 2.4 that is incompatible to the extent that you now need to declare some
> principals you didn't need to set now need to be
>
>
> 1. it Looks like HDFS-3604, enable webhdfs is the issue that stops the NN
> and DN  -IMO it should be marked as an incompatible change, as it stops
> clusters that used to start, from starting.
> 2. what's causing this in YARN?
>


Mime
View raw message