falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-1153) Falcon kill on /instance/kill/feed fails intermittently
Date Thu, 16 Apr 2015 23:12:59 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498786#comment-14498786
] 

Balu Vellanki edited comment on FALCON-1153 at 4/16/15 11:12 PM:
-----------------------------------------------------------------

[~sriksun]  I increased the default count to 30, and will pick up the count from runtime.properties
"scheduler.action.status.retry.count" if this property is defined. The delay is kept at 100ms.
Since there is no good location to document this property, the documentation for this property
will be added when https://issues.apache.org/jira/browse/FALCON-778 is completed.

[~sriksun] [~ajayydv] Attached is the new patch, please review. 


was (Author: bvellanki):
[~sriksun]  I increased the default count to 30, and will pick up the count from runtime.properties
"oozie.action.retry.count" if this property is defined. The delay is kept at 100ms. Since
there is no good location to document this property, the documentation for this property will
be added when https://issues.apache.org/jira/browse/FALCON-778 is completed.

[~sriksun] [~ajayydv] Attached is the new patch, please review. 

> Falcon kill on /instance/kill/feed fails intermittently
> -------------------------------------------------------
>
>                 Key: FALCON-1153
>                 URL: https://issues.apache.org/jira/browse/FALCON-1153
>             Project: Falcon
>          Issue Type: Bug
>          Components: webapp
>    Affects Versions: 0.6
>            Reporter: Balu Vellanki
>            Assignee: Balu Vellanki
>            Priority: Critical
>             Fix For: 0.6.1
>
>         Attachments: FALCON-1153-v1.patch, FALCON-1153.patch
>
>
> Kill operation fails with response:
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <instancesResult>
>    <status>FAILED</status>
>    <message>default/KILL</message>
>    <requestId>default/null</requestId>
>    <instances>
>       <instance>2015-03-12T15:38Z</instance>
>       <status>ERROR</status>
>       <logFile>http://MAINT22-FCN12:11000/oozie?job=0000135-150312134223103-oozie-hado-W</logFile>
>       <cluster>corp-34ce5f65</cluster>
>       <sourceCluster>corp-1b70e962</sourceCluster>
>       <startTime>2015-03-12T16:26:10Z</startTime>
>       <details />
>    </instances>
> </instancesResult>
> {code}
> Stacktrace from falcon.application.log:
> {code}
> 2015-03-12 16:26:19,506 WARN  - [941235624@qtp-2073066709-135:d88a0d21-00c0-4646-8f05-ee606c6170a5
hadoopqa:POST//instance/kill/feed/raaw-logs16-ac0c9860/] ~ Unable to perform action KILL on
cluster (OozieWorkflowEngine:577)
> org.apache.falcon.FalconException: For Job0000135-150312134223103-oozie-hado-W, actual
statuses: RUNNING, expected statuses: [KILLED, SUCCEEDED, FAILED]
> 	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.assertStatus(OozieWorkflowEngine.java:1341)
> 	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.kill(OozieWorkflowEngine.java:1424)
> 	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.performAction(OozieWorkflowEngine.java:757)
> 	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.doJobAction(OozieWorkflowEngine.java:575)
> 	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.killInstances(OozieWorkflowEngine.java:495)
> 	at org.apache.falcon.resource.AbstractInstanceManager.killInstance(AbstractInstanceManager.java:392)
> 	at org.apache.falcon.resource.InstanceManager.killInstance(InstanceManager.java:180)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
> 	at org.apache.falcon.resource.proxy.InstanceManagerProxy$8.doExecute(InstanceManagerProxy.java:265)
> 	at org.apache.falcon.resource.proxy.InstanceManagerProxy$8.doExecute(InstanceManagerProxy.java:1)
> 	at org.apache.falcon.resource.proxy.InstanceManagerProxy$InstanceProxy.execute(InstanceManagerProxy.java:357)
> 	at org.apache.falcon.resource.proxy.InstanceManagerProxy.killInstance_aroundBody14(InstanceManagerProxy.java:268)
> 	at org.apache.falcon.resource.proxy.InstanceManagerProxy$AjcClosure15.run(InstanceManagerProxy.java:1)
> 	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> 	at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
> 	at org.apache.falcon.resource.proxy.InstanceManagerProxy.killInstance(InstanceManagerProxy.java:252)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> 	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> 	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> 	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> 	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> 	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> 	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> 	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> 	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> 	at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:102)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 	at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184)
> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
> 	at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:193)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 	at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:65)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> 	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> 	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> 	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 	at org.mortbay.jetty.Server.handle(Server.java:326)
> 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> 	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> 	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message