falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-1411) El expressions not getting evaluated correctly with falcon-0.7 jar
Date Thu, 20 Aug 2015 19:02:46 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705564#comment-14705564
] 

Ajay Yadava edited comment on FALCON-1411 at 8/20/15 7:02 PM:
--------------------------------------------------------------

On debugging Jaydeep Vishwakarma found that the actual issue is that during dryRun Oozie is
actually looking for an instance path which can match to latest(0) (any instance) and it doesn't
find any and being a synchronous operation it fails. If a feed instance is created/exists
then the issue goes away. We tested this with feeds with existing instances and same consumer
process worked smoothly.



was (Author: ajayyadava):
On debugging Jaydeep Vishwakarma found that the actual issue is that during dryRun Oozie is
actually looking for an instance path which can match to latest(0) (any instance) and it doesn't
find any and being a synchronous operation it fails. If a feed instance is created/exists
then the issue goes away. We tested this with feeds with existing instances and same consumer
process worked smoothly.
That said it is not always possible to wait for feed instances to be available or bootstrap
with a dummy one and this feature will be useful in those cases.

> El expressions not getting evaluated correctly with falcon-0.7 jar
> ------------------------------------------------------------------
>
>                 Key: FALCON-1411
>                 URL: https://issues.apache.org/jira/browse/FALCON-1411
>             Project: Falcon
>          Issue Type: Bug
>          Components: oozie
>    Affects Versions: 0.7
>         Environment: QA
>            Reporter: Pragya Mittal
>            Assignee: Pallavi Rao
>
> Following El-expressions are broken :
> 1. currentWeek and lastWeek 
> While scheduling a process containing currentWeek/lastWeek el expression it gets scheduled
but instance keeps on waiting (although there are no missing dependency)
> Process -definition is :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <process name="Adda4dfb0-5eb92594" xmlns="uri:falcon:process:0.1">
>     <clusters>
>         <cluster name="Adda4dfb0-a611389d">
>             <validity start="2015-02-17T10:30Z" end="2015-02-17T10:50Z"/>
>         </cluster>
>     </clusters>
>     <parallel>1</parallel>
>     <order>FIFO</order>
>     <frequency>minutes(5)</frequency>
>     <timezone>UTC</timezone>
>     <inputs>
>         <input name="inputData" feed="Adda4dfb0-04fe993c" start="currentWeek('WED',2,15)"
end="currentWeek('WED',2,25)"/>
>     </inputs>
>     <outputs>
>         <output name="outputData" feed="Adda4dfb0-68dd6cce" instance="now(0,0)"/>
>     </outputs>
>     <properties>
>         <property name="queueName" value="default"/>
>         <property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1,
'DAY'), 'yyyy-MMM-dd')}"/>
>         <property name="user" value="${user()}"/>
>     </properties>
>     <workflow path="/tmp/falcon-regression/ELExpCurrentAndLastWeekTest/aggregator"/>
>     <retry policy="periodic" delay="minutes(3)" attempts="3"/>
>     <ACL owner="pragya" group="dataqa" permission="*"/>
> </process>
> {code}
> Oozie stack-trace is as follows :
> {code}
> 2015-08-19 12:34:57,199  WARN pool-6-thread-2 CallableQueueService$CompositeCallable
- SERVER[ip-192-168-138-200] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-150819121241197-oozie-oozi-C]
ACTION[0000002-150819121241197-oozie-oozi-C@4] exception callable [coord_action_input], E1021:
Coord Action Input Check Error: Unable to evaluate :${dataIn('inputData', 'null')}:
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: Unable
to evaluate :${dataIn('inputData', 'null')}:
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:279)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:71)
>         at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>         at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
>         at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
>         at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Unable to evaluate :${dataIn('inputData', 'null')}:
>         at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:723)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.resolveTagContents(CoordActionInputCheckXCommand.java:398)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.materializeDataProperties(CoordActionInputCheckXCommand.java:379)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.resolveCoordConfiguration(CoordActionInputCheckXCommand.java:288)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
>         ... 8 more
> Caused by: javax.servlet.jsp.el.ELException: An error occurred while evaluating function
"dataIn"
>         at org.apache.commons.el.Logger.logError(Logger.java:484)
>         at org.apache.commons.el.Logger.logError(Logger.java:545)
>         at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:174)
>         at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
>         at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
>         at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:204)
>         at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:714)
>         ... 12 more
> {code}
> Oozie coordinator.xml contains the following property :
> {code}
>                 <property>
>                     <name>falconInPaths</name>
>                     <value>${dataIn('inputData', 'null')}</value>
>                 </property>
> {code}
> 2. future and latest
> While scheduling a process containing future/latest el expression it gets submitted but
fails on scheduling.
> Falcon stack trace shows falcon is not able to communicate with oozie and gives socket
timeout exception .
> {code}
> 2015-08-19 12:31:02,527 ERROR - [1768191773@qtp-351417028-14 - 25e8c020-17ed-4d56-85e7-f5ded1f6d442:pragya:POST//entities/schedule/process/Ac21dad17-c7ef19c7]
~ Unable to schedule workflow (AbstractSchedulableEntityManager:75)
> org.apache.falcon.FalconException: Entity schedule failed for process: Ac21dad17-c7ef19c7
>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:95)
>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:72)
>         at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:130)
>         at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>         at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>         at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>         at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>         at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>         at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>         at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>         at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>         at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>         at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>         at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>         at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:106)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184)
>         at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:396)
>         at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:193)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.falcon.FalconException: IO_ERROR : java.net.SocketTimeoutException:
Read timed out
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223)
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171)
>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93)
>         ... 45 more
> Caused by: IO_ERROR : java.net.SocketTimeoutException: Read timed out
>         at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:523)
>         at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735)
>         at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
>         at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
>         at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
>         at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
>         at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
>         ... 47 more
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message