falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Grant <ja...@queeg.org>
Subject Re: Process NPE
Date Wed, 13 Aug 2014 17:52:13 GMT
I'm having the same issue and was just about to send a mail when you posted.

I've had a look at the code and the problem seems to be when the process
has no inputs. The check in getStorageType used to short circuit if there
were no inputs but that changed as part of the changes in FALCON-443 to
only short circuit if the process has no inputs and no outputs.

Below is a patch that will prevent the exception (currently untested).

diff --git
a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
index 59361e8..33f0aad 100644
--- a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
+++ b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
@@ -59,11 +59,13 @@ public final class ProcessHelper {
             return storageType;
         }

-        for (Input input : process.getInputs().getInputs()) {
-            Feed feed = EntityUtil.getEntity(EntityType.FEED,
input.getFeed());
-            storageType = FeedHelper.getStorageType(feed, cluster);
-            if (Storage.TYPE.TABLE == storageType) {
-                break;
+        if (process.getInputs() != null) {
+            for (Input input : process.getInputs().getInputs()) {
+                Feed feed = EntityUtil.getEntity(EntityType.FEED,
input.getFeed());
+                storageType = FeedHelper.getStorageType(feed, cluster);
+                if (Storage.TYPE.TABLE == storageType) {
+                    break;
+                }
             }
         }

James


On 13 August 2014 17:54, Josh Clum <joshclum@gmail.com> wrote:

> Hi, I'm getting the following NPE when i launch with "falcon entity -type
> process -schedule -name raw-cc-bp-ratio-lcms-comp-process". At the bottom
> is my process which has no inputs.
>
> If I look at
>
> https://github.com/apache/incubator-falcon/blob/branch-0.5/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> it seems that this should never happen because process.getInputs() ==
> null should
> be true and should avoid the NPE. Any thoughts?
>
> 2014-08-13 16:49:05,152 ERROR -
> [6221571@qtp-1696932027-3
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Unable to schedule workflow
> (AbstractSchedulableEntityManager:60)
> java.lang.NullPointerException
> at
>
> org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646)
>  at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606)
>  at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206)
>  at
>
> org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111)
>  at
>
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119)
> at
>
> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68)
>  at
>
> org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57)
> at
>
> org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85)
>  at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
>  at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324)
> at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
>  at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326)
> at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1)
>  at
> org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> at
>
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
>  at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
>  at
>
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
>
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>  at
>
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
>
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>  at
>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
>
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>  at
>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
>
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>  at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>  at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>  at
>
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>  at
>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>  at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>  at
>
> org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183)
> at
>
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
>  at
>
> org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
> at
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>  at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>  at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>  at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>  at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>  at
>
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>  at
>
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> at
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 2014-08-13 16:49:05,154 ERROR -
> [6221571@qtp-1696932027-3
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Failure reason
> (FalconWebException:40)
> java.lang.NullPointerException
> at
>
> org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646)
>  at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606)
>  at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206)
>  at
>
> org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142)
> at
>
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111)
>  at
>
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119)
> at
>
> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68)
>  at
>
> org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57)
> at
>
> org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85)
>  at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
>  at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324)
> at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
>  at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326)
> at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1)
>  at
> org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> at
>
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
>  at
>
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
>  at
>
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
>
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>  at
>
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
>
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>  at
>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
>
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>  at
>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
>
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>  at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>  at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>  at
>
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>  at
>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>  at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>  at
>
> org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183)
> at
>
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
>  at
>
> org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
> at
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>  at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>  at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>  at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>  at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>  at
>
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>  at
>
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> at
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 2014-08-13 16:49:05,154 ERROR -
> [6221571@qtp-1696932027-3
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Action failed: Bad Request
> Error:null
>  (FalconWebException:66)
> 2014-08-13 16:49:05,155 INFO  -
> [6221571@qtp-1696932027-3
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> a38b5528-cae0-4b07-a8fc-482661e467a9] ~ {Action:schedule,
> Dimensions:{entityType=process, colo=*,
> entityName=raw-cc-bp-ratio-lcms-comp-process}, Status: FAILED,
> Time-taken:161608565 ns} (METRIC:32)
>
> My process looks like this:
>
> <process name="raw-cc-bp-ratio-lcms-comp-process"
> xmlns="uri:falcon:process:0.1">
>
>   <tags>type=raw</tags>
>
>   <clusters>
>     <cluster name="minicluster">
>       <validity start="2014-08-13T00:00Z" end="2016-01-01T00:00Z"/>
>     </cluster>
>   </clusters>
>
>   <parallel>1</parallel>
>   <order>FIFO</order>
>   <timeout>hours(2)</timeout>
>   <frequency>days(1)</frequency>
>
>   <outputs>
>     <output name="output" feed="raw-cc-bp-ratio-lcms-comp-feed"
> instance="now(0,0)" />
>   </outputs>
>
>   <properties>
> ...
>   </properties>
>
>   <workflow name="sqoop_wf" version="2.0.0" engine="oozie"
> path="/apps/huron/ingest/standard_ingest/sqoop_wf.xml" />
>
>   <retry policy="periodic" delay="minutes(10)" attempts="3" />
>
> </process>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message