falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pragya Mittal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1758) APIs fail when oozie workflow entries are deleted
Date Wed, 20 Jan 2016 07:00:46 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108130#comment-15108130
] 

Pragya Mittal commented on FALCON-1758:
---------------------------------------

There are some concerns when scenarios like this appear :

1. The process was scheduled on two colo. Deletion happened from one colo and not from other.
Irrespective of this , since it was not a clean delete the entity was still there in prism
ConfigStore. So user can see the entity using -list option and -defintion option.

2. Whenever there are cases like these which lead to inconsistent state , falcon server fails
to restart.
{noformat}
2016-01-20 11:22:37,546 DEBUG - [main:] ~ Received request to schedule instance PROCESS/processMerlinNative/ProcessMultipleClustersTest-corp-42c964e2/2016-01-19-14-13
with sequence 53. (SchedulerService:104)
2016-01-20 11:22:37,546 DEBUG - [main:] ~ Loading instance PROCESS/processMerlinNative/ProcessMultipleClustersTest-corp-42c964e2/2016-01-19-13-21
from state. (ProcessExecutor:129)
2016-01-20 11:22:37,546 INFO  - [main:] ~ Logging in dataqa (CurrentUser:65)
2016-01-20 11:22:37,570 DEBUG - [pool-11-thread-1:] ~ Schedule conditions not met for instance
PROCESS/processMerlinNative/ProcessMultipleClustersTest-corp-42c964e2/2016-01-19-13-25. Awaiting
on PROCESS/processMerlinNative/ProcessMultipleClustersTest-corp-42c964e2 (SchedulerService:355)
2016-01-20 11:22:37,571 DEBUG - [pool-11-thread-1:] ~ Received request to run instance PROCESS/processMerlinNative/ProcessMultipleClustersTest-corp-42c964e2/2016-01-19-13-26
(SchedulerService:299)
2016-01-20 11:22:37,576 ERROR - [main:] ~ Unable to load entity : processMerlinNative (FalconExecutionService:71)
org.apache.falcon.exception.NotificationServiceException: org.apache.falcon.exception.DAGEngineException:
E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.notification.service.impl.JobCompletionService.register(JobCompletionService.java:92)
	at org.apache.falcon.notification.service.NotificationServicesRegistry.register(NotificationServicesRegistry.java:65)
	at org.apache.falcon.execution.ProcessExecutor.onSchedule(ProcessExecutor.java:500)
	at org.apache.falcon.execution.ProcessExecutor.reloadInstances(ProcessExecutor.java:132)
	at org.apache.falcon.execution.ProcessExecutor.schedule(ProcessExecutor.java:100)
	at org.apache.falcon.execution.FalconExecutionService.init(FalconExecutionService.java:68)
	at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
	at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
	at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:224)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
	at org.apache.falcon.FalconServer.main(FalconServer.java:102)
Caused by: org.apache.falcon.exception.DAGEngineException: E0604 : E0604: Job does not exist
[0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.workflow.engine.OozieDAGEngine.getConfiguration(OozieDAGEngine.java:399)
	at org.apache.falcon.notification.service.impl.JobCompletionService.register(JobCompletionService.java:84)
	... 18 more
Caused by: E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
	at org.apache.oozie.client.OozieClient$JobInfo.call(OozieClient.java:850)
	at org.apache.oozie.client.OozieClient$JobInfo.call(OozieClient.java:834)
	at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
	at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:925)
	at org.apache.oozie.client.ProxyOozieClient.access$1301(ProxyOozieClient.java:48)
	at org.apache.oozie.client.ProxyOozieClient$13.call(ProxyOozieClient.java:328)
	at org.apache.oozie.client.ProxyOozieClient$13.call(ProxyOozieClient.java:325)
	at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
	at org.apache.oozie.client.ProxyOozieClient.getJobInfo(ProxyOozieClient.java:325)
	at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:903)
	at org.apache.oozie.client.ProxyOozieClient.access$1201(ProxyOozieClient.java:48)
	at org.apache.oozie.client.ProxyOozieClient$12.call(ProxyOozieClient.java:311)
	at org.apache.oozie.client.ProxyOozieClient$12.call(ProxyOozieClient.java:308)
	at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
	at org.apache.oozie.client.ProxyOozieClient.getJobInfo(ProxyOozieClient.java:308)
	at org.apache.falcon.workflow.engine.OozieDAGEngine.getConfiguration(OozieDAGEngine.java:391)
	... 19 more
2016-01-20 11:22:37,578 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.execution.FalconExecutionService
(ServiceInitializer:49)
java.lang.RuntimeException: org.apache.falcon.exception.NotificationServiceException: org.apache.falcon.exception.DAGEngineException:
E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.execution.FalconExecutionService.init(FalconExecutionService.java:72)
	at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
	at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
	at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:224)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
	at org.apache.falcon.FalconServer.main(FalconServer.java:102)
Caused by: org.apache.falcon.exception.NotificationServiceException: org.apache.falcon.exception.DAGEngineException:
E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.notification.service.impl.JobCompletionService.register(JobCompletionService.java:92)
	at org.apache.falcon.notification.service.NotificationServicesRegistry.register(NotificationServicesRegistry.java:65)
	at org.apache.falcon.execution.ProcessExecutor.onSchedule(ProcessExecutor.java:500)
	at org.apache.falcon.execution.ProcessExecutor.reloadInstances(ProcessExecutor.java:132)
	at org.apache.falcon.execution.ProcessExecutor.schedule(ProcessExecutor.java:100)
	at org.apache.falcon.execution.FalconExecutionService.init(FalconExecutionService.java:68)
	... 13 more
Caused by: org.apache.falcon.exception.DAGEngineException: E0604 : E0604: Job does not exist
[0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.workflow.engine.OozieDAGEngine.getConfiguration(OozieDAGEngine.java:399)
	at org.apache.falcon.notification.service.impl.JobCompletionService.register(JobCompletionService.java:84)
	... 18 more
Caused by: E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
	at org.apache.oozie.client.OozieClient$JobInfo.call(OozieClient.java:850)
	at org.apache.oozie.client.OozieClient$JobInfo.call(OozieClient.java:834)
	at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
	at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:925)
	at org.apache.oozie.client.ProxyOozieClient.access$1301(ProxyOozieClient.java:48)
	at org.apache.oozie.client.ProxyOozieClient$13.call(ProxyOozieClient.java:328)
	at org.apache.oozie.client.ProxyOozieClient$13.call(ProxyOozieClient.java:325)
	at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
	at org.apache.oozie.client.ProxyOozieClient.getJobInfo(ProxyOozieClient.java:325)
	at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:903)
	at org.apache.oozie.client.ProxyOozieClient.access$1201(ProxyOozieClient.java:48)
	at org.apache.oozie.client.ProxyOozieClient$12.call(ProxyOozieClient.java:311)
	at org.apache.oozie.client.ProxyOozieClient$12.call(ProxyOozieClient.java:308)
	at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
	at org.apache.oozie.client.ProxyOozieClient.getJobInfo(ProxyOozieClient.java:308)
	at org.apache.falcon.workflow.engine.OozieDAGEngine.getConfiguration(OozieDAGEngine.java:391)
	... 19 more
2016-01-20 11:22:37,578 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@23ca36d{/,/mnt/falcon/server/server/webapp/falcon}
(log:87)
java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException:
org.apache.falcon.exception.NotificationServiceException: org.apache.falcon.exception.DAGEngineException:
E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
	at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:224)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
	at org.apache.falcon.FalconServer.main(FalconServer.java:102)
Caused by: org.apache.falcon.FalconException: java.lang.RuntimeException: org.apache.falcon.exception.NotificationServiceException:
org.apache.falcon.exception.DAGEngineException: E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:50)
	at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
	... 11 more
Caused by: java.lang.RuntimeException: org.apache.falcon.exception.NotificationServiceException:
org.apache.falcon.exception.DAGEngineException: E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.execution.FalconExecutionService.init(FalconExecutionService.java:72)
	at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
	... 12 more
Caused by: org.apache.falcon.exception.NotificationServiceException: org.apache.falcon.exception.DAGEngineException:
E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.notification.service.impl.JobCompletionService.register(JobCompletionService.java:92)
	at org.apache.falcon.notification.service.NotificationServicesRegistry.register(NotificationServicesRegistry.java:65)
	at org.apache.falcon.execution.ProcessExecutor.onSchedule(ProcessExecutor.java:500)
	at org.apache.falcon.execution.ProcessExecutor.reloadInstances(ProcessExecutor.java:132)
	at org.apache.falcon.execution.ProcessExecutor.schedule(ProcessExecutor.java:100)
	at org.apache.falcon.execution.FalconExecutionService.init(FalconExecutionService.java:68)
	... 13 more
Caused by: org.apache.falcon.exception.DAGEngineException: E0604 : E0604: Job does not exist
[0000973-160119150246727-oozie-oozi-W]
	at org.apache.falcon.workflow.engine.OozieDAGEngine.getConfiguration(OozieDAGEngine.java:399)
	at org.apache.falcon.notification.service.impl.JobCompletionService.register(JobCompletionService.java:84)
	... 18 more
Caused by: E0604 : E0604: Job does not exist [0000973-160119150246727-oozie-oozi-W]
	at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
	at org.apache.oozie.client.OozieClient$JobInfo.call(OozieClient.java:850)
	at org.apache.oozie.client.OozieClient$JobInfo.call(OozieClient.java:834)
	at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
	at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:925)
	at org.apache.oozie.client.ProxyOozieClient.access$1301(ProxyOozieClient.java:48)
	at org.apache.oozie.client.ProxyOozieClient$13.call(ProxyOozieClient.java:328)
	at org.apache.oozie.client.ProxyOozieClient$13.call(ProxyOozieClient.java:325)
	at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
	at org.apache.oozie.client.ProxyOozieClient.getJobInfo(ProxyOozieClient.java:325)
	at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:903)
	at org.apache.oozie.client.ProxyOozieClient.access$1201(ProxyOozieClient.java:48)
	at org.apache.oozie.client.ProxyOozieClient$12.call(ProxyOozieClient.java:311)
	at org.apache.oozie.client.ProxyOozieClient$12.call(ProxyOozieClient.java:308)
	at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
	at org.apache.oozie.client.ProxyOozieClient.getJobInfo(ProxyOozieClient.java:308)
	at org.apache.falcon.workflow.engine.OozieDAGEngine.getConfiguration(OozieDAGEngine.java:391)
	... 19 more

{noformat}

> APIs fail when oozie workflow entries are deleted
> -------------------------------------------------
>
>                 Key: FALCON-1758
>                 URL: https://issues.apache.org/jira/browse/FALCON-1758
>             Project: Falcon
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 0.9
>            Reporter: Pragya Mittal
>            Assignee: pavan kumar kolamuri
>             Fix For: 0.9
>
>
> Whenever a process is scheduled in Falcon Native Scheduler and instances are running
, later if those workflow entries got deleted , then entity deletion fails with following
exception
> {noformat}
> 2016-01-19 14:41:42,261 INFO  - [ActiveMQ Session Task-56:] ~ Logging in pragya (CurrentUser:65)
> 2016-01-19 14:41:42,261 INFO  - [ActiveMQ Session Task-56:] ~ Creating Oozie client object
for http://192.168.138.236:11000/oozie/ (OozieClientFactory:50)
> 2016-01-19 14:41:42,267 DEBUG - [ActiveMQ Session Task-56:] ~ Error while retrieving
JMS connection info (OozieWorkflowEngine:613)
> E1601 : E1601: Cannot retrieve JMS connection info [JMSTopicService is not initialized.
JMS notificationmay not be enabled]
>         at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
>         at org.apache.oozie.client.OozieClient$JMSInfo.call(OozieClient.java:869)
>         at org.apache.oozie.client.OozieClient$JMSInfo.call(OozieClient.java:856)
>         at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
>         at org.apache.oozie.client.OozieClient.getJMSConnectionInfo(OozieClient.java:912)
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.isNotificationEnabled(OozieWorkflowEngine.java:604)
>         at org.apache.falcon.workflow.WorkflowJobEndNotificationService.notifyWorkflowEnd(WorkflowJobEndNotificationService.java:214)
>         at org.apache.falcon.workflow.WorkflowJobEndNotificationService.notifySuccess(WorkflowJobEndNotificationService.java:105)
>         at org.apache.falcon.messaging.JMSMessageConsumer.invokeListener(JMSMessageConsumer.java:218)
>         at org.apache.falcon.messaging.JMSMessageConsumer.onMessage(JMSMessageConsumer.java:114)
>         at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1393)
>         at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
>         at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
>         at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
>         at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 2016-01-19 14:41:42,267 INFO  - [ActiveMQ Session Task-56:] ~ Creating Oozie client object
for http://192.168.138.236:11000/oozie/ (OozieClientFactory:50)
> {noformat}
> However if the same is done for entities scheduled with oozie scheduler, entity deletion
happens successfully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message