hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tallanel Siriel <tallanel.had...@gmail.com>
Subject Yarn REST API - cannot kill an application
Date Tue, 05 Apr 2016 09:52:36 GMT
Hello !

I send you this mail because I have a problem when I try to kill an
application I just submit.
I have Hadoop 2.7.1 (HDP 2.3.2).

Pre-requisite :
- My Resource Manager is in HA
- My cluster is Kerberized

Execution of a job (I wordcount a logfile, it does not really matter) :
###
hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount -Dmapreduce.job.queuename=<a yarn queue> /tmp/ambari-server.log.9
/tmp/myresult/
###

While the job is running, I try to kill it :
###
curl --negotiate -ikv -u: -X POST -d '{"state": "KILLED"}' 'http://<ACTIVE
RM>:8088/ws/v1/cluster/apps/<APPLICATION_ID>'
###

And here is the log :
###
* About to connect() to <ACTIVE RM> port 8088 (#0)
* Trying <ACTIVE_RM_IP>... connected
* Connected to <ACTIVE RM> (<ACTIVE_RM_IP>) port 8088 (#0)
> POST /ws/v1/cluster/apps/<APPLICATION_ID> HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/
3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <ACTIVE RM>:8088
> Accept: */*
> Content-Length: 19
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Tue, 05 Apr 2016 08:56:30 GMT
Expires: Tue, 05 Apr 2016 08:56:30 GMT
< Date: Tue, 05 Apr 2016 08:56:30 GMT
Date: Tue, 05 Apr 2016 08:56:30 GMT
< Pragma: no-cache
Pragma: no-cache
< Expires: Tue, 05 Apr 2016 08:56:30 GMT
Expires: Tue, 05 Apr 2016 08:56:30 GMT
< Date: Tue, 05 Apr 2016 08:56:30 GMT
Date: Tue, 05 Apr 2016 08:56:30 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: application/xml
Content-Type: application/xml
< Content-Length: 202
Content-Length: 202
< Server: Jetty(6.1.26.hwx)
Server: Jetty(6.1.26.hwx)

<
* Connection #0 to host <ACTIVE RM> left intact
* Closing connection #0
<?xml version="1.0" encoding="UTF-8"
standalone="yes"?><RemoteException><exception>WebApplicationException</exception><javaClassName>javax.ws.rs.WebApplicationException</javaClassName></RemoteException>
###

And in the logs of the RM, I can see the following :
###
2016-04-05 11:05:05,193 INFO  rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(417)) -
container_e36_1459845370345_0006_01_000002 Container Transitioned from
ACQUIRED to RUNNING
2016-04-05 11:05:05,518 INFO  scheduler.AppSchedulingInfo
(AppSchedulingInfo.java:updateResourceRequests(185)) - checking for
deactivate of application : <APPLICATION_ID>
2016-04-05 11:05:07,733 WARN  webapp.GenericExceptionHandler
(GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
javax.ws.rs.WebApplicationException
        at
com.sun.jersey.server.impl.uri.rules.TerminatingRule.accept(TerminatingRule.java:66)
        at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
        at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
        at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:142)
        at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
        at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
        at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-04-05 11:05:16,274 INFO  rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(417)) -
container_e36_1459845370345_0006_01_000002 Container Transitioned from
RUNNING to COMPLETED
###

Does anyone has this problem with HA RM ?
I noticed that I can normally kill the application using Yarn CLI, but I
cannot using the Yarn REST API.

BR.

Tallanel.

Mime
View raw message