falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaydeep Vishwakarma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1411) El expressions not getting evaluated correctly with falcon-0.7 jar
Date Thu, 27 Aug 2015 11:52:46 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716521#comment-14716521
] 

Jaydeep Vishwakarma commented on FALCON-1411:
---------------------------------------------

Yes, dryrun does not wait for input data, But it does synchronous call. Everytime it generates
the new path by jobs frequency and it checks the same path in hdfs till it find the path or
it reaches to initial instance.
When you have initial instance very old and path does not exist or belongs to very old instance
, you might experience the slowness. 

> El expressions not getting evaluated correctly with falcon-0.7 jar
> ------------------------------------------------------------------
>
>                 Key: FALCON-1411
>                 URL: https://issues.apache.org/jira/browse/FALCON-1411
>             Project: Falcon
>          Issue Type: Bug
>          Components: oozie
>    Affects Versions: 0.7
>         Environment: QA
>            Reporter: Pragya Mittal
>            Assignee: Pallavi Rao
>
> Following El-expressions are broken :
> 1. currentWeek and lastWeek 
> While scheduling a process containing currentWeek/lastWeek el expression it gets scheduled
but instance keeps on waiting (although there are no missing dependency)
> Process -definition is :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <process name="Adda4dfb0-5eb92594" xmlns="uri:falcon:process:0.1">
>     <clusters>
>         <cluster name="Adda4dfb0-a611389d">
>             <validity start="2015-02-17T10:30Z" end="2015-02-17T10:50Z"/>
>         </cluster>
>     </clusters>
>     <parallel>1</parallel>
>     <order>FIFO</order>
>     <frequency>minutes(5)</frequency>
>     <timezone>UTC</timezone>
>     <inputs>
>         <input name="inputData" feed="Adda4dfb0-04fe993c" start="currentWeek('WED',2,15)"
end="currentWeek('WED',2,25)"/>
>     </inputs>
>     <outputs>
>         <output name="outputData" feed="Adda4dfb0-68dd6cce" instance="now(0,0)"/>
>     </outputs>
>     <properties>
>         <property name="queueName" value="default"/>
>         <property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1,
'DAY'), 'yyyy-MMM-dd')}"/>
>         <property name="user" value="${user()}"/>
>     </properties>
>     <workflow path="/tmp/falcon-regression/ELExpCurrentAndLastWeekTest/aggregator"/>
>     <retry policy="periodic" delay="minutes(3)" attempts="3"/>
>     <ACL owner="pragya" group="dataqa" permission="*"/>
> </process>
> {code}
> Oozie stack-trace is as follows :
> {code}
> 2015-08-19 12:34:57,199  WARN pool-6-thread-2 CallableQueueService$CompositeCallable
- SERVER[ip-192-168-138-200] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-150819121241197-oozie-oozi-C]
ACTION[0000002-150819121241197-oozie-oozi-C@4] exception callable [coord_action_input], E1021:
Coord Action Input Check Error: Unable to evaluate :${dataIn('inputData', 'null')}:
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: Unable
to evaluate :${dataIn('inputData', 'null')}:
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:279)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:71)
>         at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>         at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
>         at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
>         at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Unable to evaluate :${dataIn('inputData', 'null')}:
>         at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:723)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.resolveTagContents(CoordActionInputCheckXCommand.java:398)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.materializeDataProperties(CoordActionInputCheckXCommand.java:379)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.resolveCoordConfiguration(CoordActionInputCheckXCommand.java:288)
>         at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
>         ... 8 more
> Caused by: javax.servlet.jsp.el.ELException: An error occurred while evaluating function
"dataIn"
>         at org.apache.commons.el.Logger.logError(Logger.java:484)
>         at org.apache.commons.el.Logger.logError(Logger.java:545)
>         at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:174)
>         at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
>         at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
>         at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:204)
>         at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:714)
>         ... 12 more
> {code}
> Oozie coordinator.xml contains the following property :
> {code}
>                 <property>
>                     <name>falconInPaths</name>
>                     <value>${dataIn('inputData', 'null')}</value>
>                 </property>
> {code}
> 2. future and latest
> While scheduling a process containing future/latest el expression it gets submitted but
fails on scheduling.
> Falcon stack trace shows falcon is not able to communicate with oozie and gives socket
timeout exception .
> {code}
> 2015-08-19 12:31:02,527 ERROR - [1768191773@qtp-351417028-14 - 25e8c020-17ed-4d56-85e7-f5ded1f6d442:pragya:POST//entities/schedule/process/Ac21dad17-c7ef19c7]
~ Unable to schedule workflow (AbstractSchedulableEntityManager:75)
> org.apache.falcon.FalconException: Entity schedule failed for process: Ac21dad17-c7ef19c7
>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:95)
>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:72)
>         at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:130)
>         at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>         at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>         at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>         at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>         at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>         at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>         at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>         at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>         at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>         at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>         at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>         at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:106)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184)
>         at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:396)
>         at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:193)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.falcon.FalconException: IO_ERROR : java.net.SocketTimeoutException:
Read timed out
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223)
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171)
>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93)
>         ... 45 more
> Caused by: IO_ERROR : java.net.SocketTimeoutException: Read timed out
>         at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:523)
>         at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735)
>         at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
>         at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
>         at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
>         at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
>         at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
>         ... 47 more
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message