falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balu Vellanki <b...@hortonworks.com>
Subject Re: Process NPE
Date Wed, 13 Aug 2014 22:33:04 GMT
A JIRA https://issues.apache.org/jira/browse/FALCON-598 was opened by David
Kjerrumgaard for this issue. I am working on a patch now.

Thanks
Balu Vellanki


On Wed, Aug 13, 2014 at 10:52 AM, James Grant <james@queeg.org> wrote:

> I'm having the same issue and was just about to send a mail when you
> posted.
>
> I've had a look at the code and the problem seems to be when the process
> has no inputs. The check in getStorageType used to short circuit if there
> were no inputs but that changed as part of the changes in FALCON-443 to
> only short circuit if the process has no inputs and no outputs.
>
> Below is a patch that will prevent the exception (currently untested).
>
> diff --git
> a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> index 59361e8..33f0aad 100644
> --- a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> +++ b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> @@ -59,11 +59,13 @@ public final class ProcessHelper {
>              return storageType;
>          }
>
> -        for (Input input : process.getInputs().getInputs()) {
> -            Feed feed = EntityUtil.getEntity(EntityType.FEED,
> input.getFeed());
> -            storageType = FeedHelper.getStorageType(feed, cluster);
> -            if (Storage.TYPE.TABLE == storageType) {
> -                break;
> +        if (process.getInputs() != null) {
> +            for (Input input : process.getInputs().getInputs()) {
> +                Feed feed = EntityUtil.getEntity(EntityType.FEED,
> input.getFeed());
> +                storageType = FeedHelper.getStorageType(feed, cluster);
> +                if (Storage.TYPE.TABLE == storageType) {
> +                    break;
> +                }
>              }
>          }
>
> James
>
>
> On 13 August 2014 17:54, Josh Clum <joshclum@gmail.com> wrote:
>
> > Hi, I'm getting the following NPE when i launch with "falcon entity -type
> > process -schedule -name raw-cc-bp-ratio-lcms-comp-process". At the bottom
> > is my process which has no inputs.
> >
> > If I look at
> >
> >
> https://github.com/apache/incubator-falcon/blob/branch-0.5/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> > it seems that this should never happen because process.getInputs() ==
> > null should
> > be true and should avoid the NPE. Any thoughts?
> >
> > 2014-08-13 16:49:05,152 ERROR -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Unable to schedule workflow
> > (AbstractSchedulableEntityManager:60)
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111)
> >  at
> >
> >
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119)
> > at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68)
> >  at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57)
> > at
> >
> >
> org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85)
> >  at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >  at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1)
> >  at
> > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> > at
> >
> >
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310)
> > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >  at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> >  at
> >
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> > at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> >  at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >  at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183)
> > at
> >
> >
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >  at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >  at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >  at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >  at org.mortbay.jetty.Server.handle(Server.java:326)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >  at
> >
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >  at
> >
> >
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> > at
> >
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > 2014-08-13 16:49:05,154 ERROR -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Failure reason
> > (FalconWebException:40)
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111)
> >  at
> >
> >
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119)
> > at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68)
> >  at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57)
> > at
> >
> >
> org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85)
> >  at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >  at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1)
> >  at
> > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> > at
> >
> >
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310)
> > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >  at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> >  at
> >
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> > at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> >  at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >  at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183)
> > at
> >
> >
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >  at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >  at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >  at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >  at org.mortbay.jetty.Server.handle(Server.java:326)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >  at
> >
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >  at
> >
> >
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> > at
> >
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > 2014-08-13 16:49:05,154 ERROR -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Action failed: Bad Request
> > Error:null
> >  (FalconWebException:66)
> > 2014-08-13 16:49:05,155 INFO  -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ {Action:schedule,
> > Dimensions:{entityType=process, colo=*,
> > entityName=raw-cc-bp-ratio-lcms-comp-process}, Status: FAILED,
> > Time-taken:161608565 ns} (METRIC:32)
> >
> > My process looks like this:
> >
> > <process name="raw-cc-bp-ratio-lcms-comp-process"
> > xmlns="uri:falcon:process:0.1">
> >
> >   <tags>type=raw</tags>
> >
> >   <clusters>
> >     <cluster name="minicluster">
> >       <validity start="2014-08-13T00:00Z" end="2016-01-01T00:00Z"/>
> >     </cluster>
> >   </clusters>
> >
> >   <parallel>1</parallel>
> >   <order>FIFO</order>
> >   <timeout>hours(2)</timeout>
> >   <frequency>days(1)</frequency>
> >
> >   <outputs>
> >     <output name="output" feed="raw-cc-bp-ratio-lcms-comp-feed"
> > instance="now(0,0)" />
> >   </outputs>
> >
> >   <properties>
> > ...
> >   </properties>
> >
> >   <workflow name="sqoop_wf" version="2.0.0" engine="oozie"
> > path="/apps/huron/ingest/standard_ingest/sqoop_wf.xml" />
> >
> >   <retry policy="periodic" delay="minutes(10)" attempts="3" />
> >
> > </process>
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message