Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0A7D711770 for ; Wed, 13 Aug 2014 17:52:44 +0000 (UTC) Received: (qmail 79987 invoked by uid 500); 13 Aug 2014 17:52:43 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 79946 invoked by uid 500); 13 Aug 2014 17:52:43 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 79935 invoked by uid 99); 13 Aug 2014 17:52:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 17:52:43 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.82.173] (HELO mail-we0-f173.google.com) (74.125.82.173) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 17:52:15 +0000 Received: by mail-we0-f173.google.com with SMTP id q58so82997wes.4 for ; Wed, 13 Aug 2014 10:52:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=EbiTUJyR40YZcfKhkJHWW6m2cZaeCpRjvVfU3s5GfUk=; b=G/lS483Hk9VdlDqNFOboZTnOWBi9qfL11d2hFmtNgKnfmMcBepd5A/rQ//7nP+aVer gGtXCHTrLeehInAqOaI2ISbk8tUlaUQkEEJyi+v7TxshJZg3kMtEAzi2UzuWgNaE0B7h 0ks1rrXxUNLW5BtzGVdBC0GS3Ryek5xqFvM8Dyx9lRyKn+45Q214RujYfxY1OwMrzHWd TBgbkl8Wyu7vTqAlbSZrht9oFzCgzevIQqXihq8EUhOGETosN1xtxscx6SDwbvBVWsjJ a8To9RvzvNBQu7mzCUrlZ38rkZATUYtqZDgS4Zi+t4r9dD7Q+ZcjttGTpwWcGQixGs6m /hqQ== X-Gm-Message-State: ALoCoQnXcRSkrHGoymS7G9fCFikfgF0SRmuE1qbLOedWr7wxb5TlkFPKQWagFczUJ8kIYoIRxk1w MIME-Version: 1.0 X-Received: by 10.194.179.73 with SMTP id de9mr5785580wjc.87.1407952334050; Wed, 13 Aug 2014 10:52:14 -0700 (PDT) Received: by 10.180.82.227 with HTTP; Wed, 13 Aug 2014 10:52:13 -0700 (PDT) In-Reply-To: References: Date: Wed, 13 Aug 2014 18:52:13 +0100 Message-ID: Subject: Re: Process NPE From: James Grant To: dev@falcon.incubator.apache.org Content-Type: multipart/alternative; boundary=089e013c63a6bb6ddc05008672bf X-Virus-Checked: Checked by ClamAV on apache.org --089e013c63a6bb6ddc05008672bf Content-Type: text/plain; charset=UTF-8 I'm having the same issue and was just about to send a mail when you posted. I've had a look at the code and the problem seems to be when the process has no inputs. The check in getStorageType used to short circuit if there were no inputs but that changed as part of the changes in FALCON-443 to only short circuit if the process has no inputs and no outputs. Below is a patch that will prevent the exception (currently untested). diff --git a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java index 59361e8..33f0aad 100644 --- a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java +++ b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java @@ -59,11 +59,13 @@ public final class ProcessHelper { return storageType; } - for (Input input : process.getInputs().getInputs()) { - Feed feed = EntityUtil.getEntity(EntityType.FEED, input.getFeed()); - storageType = FeedHelper.getStorageType(feed, cluster); - if (Storage.TYPE.TABLE == storageType) { - break; + if (process.getInputs() != null) { + for (Input input : process.getInputs().getInputs()) { + Feed feed = EntityUtil.getEntity(EntityType.FEED, input.getFeed()); + storageType = FeedHelper.getStorageType(feed, cluster); + if (Storage.TYPE.TABLE == storageType) { + break; + } } } James On 13 August 2014 17:54, Josh Clum wrote: > Hi, I'm getting the following NPE when i launch with "falcon entity -type > process -schedule -name raw-cc-bp-ratio-lcms-comp-process". At the bottom > is my process which has no inputs. > > If I look at > > https://github.com/apache/incubator-falcon/blob/branch-0.5/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java > it seems that this should never happen because process.getInputs() == > null should > be true and should avoid the NPE. Any thoughts? > > 2014-08-13 16:49:05,152 ERROR - > [6221571@qtp-1696932027-3 > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Unable to schedule workflow > (AbstractSchedulableEntityManager:60) > java.lang.NullPointerException > at > > org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206) > at > > org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111) > at > > org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119) > at > > org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68) > at > > org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57) > at > > org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85) > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1) > at > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > at > > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310) > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > at > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > at > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > at > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > > org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183) > at > > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392) > at > > org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221) > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) > at > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > 2014-08-13 16:49:05,154 ERROR - > [6221571@qtp-1696932027-3 > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Failure reason > (FalconWebException:40) > java.lang.NullPointerException > at > > org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206) > at > > org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142) > at > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111) > at > > org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119) > at > > org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68) > at > > org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57) > at > > org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85) > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1) > at > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > at > > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50) > at > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310) > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > at > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > at > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > at > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > > org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183) > at > > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392) > at > > org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221) > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) > at > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > 2014-08-13 16:49:05,154 ERROR - > [6221571@qtp-1696932027-3 > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Action failed: Bad Request > Error:null > (FalconWebException:66) > 2014-08-13 16:49:05,155 INFO - > [6221571@qtp-1696932027-3 > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ {Action:schedule, > Dimensions:{entityType=process, colo=*, > entityName=raw-cc-bp-ratio-lcms-comp-process}, Status: FAILED, > Time-taken:161608565 ns} (METRIC:32) > > My process looks like this: > > xmlns="uri:falcon:process:0.1"> > > type=raw > > > > > > > > 1 > FIFO > hours(2) > days(1) > > > instance="now(0,0)" /> > > > > ... > > > path="/apps/huron/ingest/standard_ingest/sqoop_wf.xml" /> > > > > > --089e013c63a6bb6ddc05008672bf--