falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shwetha GS <shwetha...@inmobi.com>
Subject Re: Need help for the problem that i faced
Date Thu, 27 Nov 2014 06:40:28 GMT
Try the java action in oozie:
http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a3.2.7_Java_Action

On Thu, Nov 27, 2014 at 11:49 AM, Gaurav Agarwal <gzagarwal@gmail.com>
wrote:

> Shwetha
>
> Apologise to trouble everyone but i am new to this falcon. Do i need to
> check the logs in JobBrowser?
>
> there is stderror
>
> Log Length: 188
> Error: Could not find or load main class com.jpmc.cib.is.bigdata.
> falcon.SemsStaticDataProvider
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.ShellMain],
> exit code [1]
>
>
>
>
>
> On Thu, Nov 27, 2014 at 11:43 AM, Shwetha GS <shwetha.gs@inmobi.com>
> wrote:
>
> > oozie launches a hadoop job which executes the shell script. Check the
> > launcher job logs
> >
> > On Thu, Nov 27, 2014 at 11:39 AM, Gaurav Agarwal <gzagarwal@gmail.com>
> > wrote:
> >
> > > Srikanth
> > >
> > > What do we mean by that in oozie context?
> > >
> > > Thanks
> > >
> > > On Thu, Nov 27, 2014 at 11:36 AM, Srikanth Sundarrajan <
> > > sriksun@hotmail.com>
> > > wrote:
> > >
> > > > Looks like the script ingest.sh is returning non-zero exit code.
> > > >
> > > > Regards
> > > > Srikanth Sundarrajan
> > > >
> > > > > Date: Thu, 27 Nov 2014 11:31:30 +0530
> > > > > Subject: Re: Need help for the problem that i faced
> > > > > From: gzagarwal@gmail.com
> > > > > To: dev@falcon.incubator.apache.org
> > > > >
> > > > > These are the log
> > > > >
> > > > > 2014-11-27 05:37:02,726 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@:start:] Start action
> > > > > [0000129-141126162609456-oozie-oozi-W@:start:] with user-retry
> > state :
> > > > > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> > > > > 2014-11-27 05:37:02,727 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@:start:]
> > > > > [***0000129-141126162609456-oozie-oozi-W@:start:***]Action
> > status=DONE
> > > > > 2014-11-27 05:37:02,727 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@:start:]
> > > > > [***0000129-141126162609456-oozie-oozi-W@:start:***]Action updated
> > in
> > > > DB!
> > > > > 2014-11-27 05:37:03,151 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] Start
> action
> > > > > [0000129-141126162609456-oozie-oozi-W@shell-node] with user-retry
> > > state
> > > > :
> > > > > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> > > > > 2014-11-27 05:37:05,377 INFO ShellActionExecutor:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] checking
> > > action,
> > > > > hadoop job ID [job_1417019189193_0117] status [RUNNING]
> > > > > 2014-11-27 05:37:05,378 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node]
> > > > > [***0000129-141126162609456-oozie-oozi-W@shell-node***]Action
> > > > > status=RUNNING
> > > > > 2014-11-27 05:37:05,379 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node]
> > > > > [***0000129-141126162609456-oozie-oozi-W@shell-node***]Action
> > updated
> > > in
> > > > > DB!
> > > > > 2014-11-27 05:37:49,260 INFO CallbackServlet:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] callback
> for
> > > > action
> > > > > [0000129-141126162609456-oozie-oozi-W@shell-node]
> > > > > 2014-11-27 05:37:50,753 INFO ShellActionExecutor:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] action
> > > > completed,
> > > > > external ID [job_1417019189193_0117]
> > > > > 2014-11-27 05:37:50,761 WARN ShellActionExecutor:546 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] Launcher
> > > ERROR,
> > > > > reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit
> > > code
> > > > > [1]
> > > > > 2014-11-27 05:37:50,865 INFO ActionEndXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] ERROR is
> > > > considered
> > > > > as FAILED for SLA
> > > > > 2014-11-27 05:37:50,941 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@fail] Start action
> > > > > [0000129-141126162609456-oozie-oozi-W@fail] with user-retry state
> :
> > > > > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> > > > > 2014-11-27 05:37:50,942 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@fail]
> > > > > [***0000129-141126162609456-oozie-oozi-W@fail***]Action
> status=DONE
> > > > > 2014-11-27 05:37:50,942 INFO ActionStartXCommand:543 - SERVER[
> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
> > > APP[shell-wf]
> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@fail]
> > > > > [***0000129-141126162609456-oozie-oozi-W@fail***]Action updated in
> > DB!
> > > > >
> > > > > On Thu, Nov 27, 2014 at 10:25 AM, Shwetha GS <
> shwetha.gs@inmobi.com>
> > > > wrote:
> > > > >
> > > > > >   <exec>ingest.sh</exec>
> > > > > > Try using absolute path.
> > > > > >
> > > > > > None of the logs imply any error. Can you attach all the oozie
> logs
> > > > for the
> > > > > > workflow?
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 27, 2014 at 7:50 AM, Gaurav Agarwal <
> > gzagarwal@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello I am running apache falcon and oozie with shell script
to
> > run
> > > > the
> > > > > > > workflow.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > getting this error
> > > > > > >
> > > > > > > *ell-node] Launcher ERROR, reason: Main class
> > > > > > > [org.apache.oozie.action.hadoop.ShellMain], exit code [1]*
> > > > > > > *2014-11-26 17:24:06,645  INFO ActionEndXCommand:543 -
> > > > > > > SERVER[sandbox.hortonworks.com <http://sandbox.hortonworks.com
> >]
> > > > > > > USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf]
> > > > > > > JOB[0000020-141126162609456-oozie-oozi-W]
> > > > > > > ACTION[0000020-141126162609456-oozie-oozi-W@shell-node]
ERROR
> is
> > > > > > > considered
> > > > > > > as FAILED for SLA*
> > > > > > > *2014-11-26 17:24:06,702  INFO ActionStartXCommand:543
-
> > > > > > > SERVER[sandbox.hortonworks.com <http://sandbox.hortonworks.com
> >]
> > > > > > > USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf]
> > > > > > > JOB[0000020-141126162609456-oozie-oozi-W]
> > > > > > > ACTION[0000020-141126162609456-oozie-oozi-W@fail] Start
action
> > > > > > > [0000020-141126162609456-oozie-*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Workflow*
> > > > > > >
> > > > > > > <!--
> > > > > > >   Licensed to the Apache Software Foundation (ASF) under
one
> > > > > > >   or more contributor license agreements.  See the NOTICE
file
> > > > > > >   distributed with this work for additional information
> > > > > > >   regarding copyright ownership.  The ASF licenses this
file
> > > > > > >   to you under the Apache License, Version 2.0 (the
> > > > > > >   "License"); you may not use this file except in compliance
> > > > > > >   with the License.  You may obtain a copy of the License
at
> > > > > > >
> > > > > > >        http://www.apache.org/licenses/LICENSE-2.0
> > > > > > >
> > > > > > >   Unless required by applicable law or agreed to in writing,
> > > software
> > > > > > >   distributed under the License is distributed on an "AS
IS"
> > BASIS,
> > > > > > >   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express
> or
> > > > > > implied.
> > > > > > >   See the License for the specific language governing
> permissions
> > > and
> > > > > > >   limitations under the License.
> > > > > > > -->
> > > > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">
> > > > > > >     <start to="shell-node"/>
> > > > > > >     <action name="shell-node">
> > > > > > >         <shell xmlns="uri:oozie:shell-action:0.2">
> > > > > > >             <job-tracker>${jobTracker}</job-tracker>
> > > > > > >             <name-node>${nameNode}</name-node>
> > > > > > >             <configuration>
> > > > > > >                 <property>
> > > > > > >                     <name>mapred.job.queue.name</name>
> > > > > > >                     <value>${queueName}</value>
> > > > > > >                 </property>
> > > > > > >             </configuration>
> > > > > > >             <exec>ingest.sh</exec>
> > > > > > >             <argument>${feedInstancePaths}</argument>
> > > > > > >             <file>${wf:appPath()}/ingest.sh#ingest.sh</file>
> > > > > > >             <!-- <file>/tmp/ingest.sh#ingest.sh</file>
-->
> > > > > > >             <!-- <capture-output/> -->
> > > > > > >         </shell>
> > > > > > >         <ok to="end"/>
> > > > > > >         <error to="fail"/>
> > > > > > >     </action>
> > > > > > >     <kill name="fail">
> > > > > > >         <message>Shell action failed, error
> > > > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > > >     </kill>
> > > > > > >     <end name="end"/>
> > > > > > > </workflow-app>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This is the shell script
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > #!/bin/bash
> > > > > > >
> > > > > > >
> > > > > > > java -cp
> > > > > > >
> > > > > >
> > > >
> > >
> >
> "/media/sf_ToSandbox/BigData-0.0.1-SNAPSHOT.jar:/usr/hdp/2.2.0.0-913/sqoop/lib/*:/usr/hdp/2.2.0.0-913/hadoop/lib/*"
> > > > > > > com.bigdata.falcon.StaticDataProvider
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > These are the feed and process and cluster information
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > <feed description="Raw customer email feed"
> > > name="FINALrawEmailFeed"
> > > > > > >     xmlns="uri:falcon:feed:0.1">
> > > > > > >
> > > > > > >
> > > >  <tags>externalSystem=USWestEmailServers,classification=secure</tags>
> > > > > > >     <groups>churnAnalysisDataPipeline</groups>
> > > > > > >     <frequency>minutes(3)</frequency>
> > > > > > >     <late-arrival cut-off="hours(4)"/>
> > > > > > >     <clusters>
> > > > > > >         <cluster name="SprimaryCluster" type="source">
> > > > > > >             <validity start="2014-11-21T00:00Z"
> > > > end="2016-03-31T00:00Z"/>
> > > > > > >             <retention limit="days(90)" action="delete"/>
> > > > > > >         </cluster>
> > > > > > >     </clusters>
> > > > > > >
> > > > > > >     <locations>
> > > > > > >         <location type="data"
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> path="/user/gaurav123/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> > > > > > >         <location type="stats" path="/none"/>
> > > > > > >         <location type="meta" path="/none"/>
> > > > > > >     </locations>
> > > > > > >
> > > > > > >     <ACL owner="ambari-qa" group="users" permission="0777"/>
> > > > > > >     <schema location="/none" provider="none"/>
> > > > > > > </feed>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Process*
> > > > > > >
> > > > > > > *-----------*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > > > > <!--
> > > > > > >  Hourly process, ingests raw emails from all US west email
> > servers
> > > > > > >  -->
> > > > > > > <process name="IngestProcess" xmlns="uri:falcon:process:0.1">
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> <tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup,externalSystem=USWestEmailServers</tags>
> > > > > > >
> > > > > > >
> > > > > > >     <clusters>
> > > > > > >         <cluster name="SprimaryCluster">
> > > > > > >             <validity start="2014-11-24T09:35Z"
> > > > end="2014-11-25T10:00Z"/>
> > > > > > >         </cluster>
> > > > > > >     </clusters>
> > > > > > >
> > > > > > >     <parallel>1</parallel>
> > > > > > >     <order>FIFO</order>
> > > > > > >     <frequency>minutes(3)</frequency>
> > > > > > >
> > > > > > >     <outputs>
> > > > > > >         <output name="output" feed="FINALrawEmailFeed"
> > > > > > instance="now(0,0)"
> > > > > > > />
> > > > > > >     </outputs>
> > > > > > >
> > > > > > >     <workflow name="IngestWorkflow" version="2.0.0"
> > > > > > >     engine="oozie"
> > > path="/user/gaurav123/falcon/demo/apps/ingest/fs"
> > > > />
> > > > > > >
> > > > > > >     <retry policy="periodic" delay="minutes(15)" attempts="3"
> />
> > > > > > >
> > > > > > > </process>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Can you please let me know what is the issue , I have tried
> since
> > > > long
> > > > > > two
> > > > > > > days? but unable to run the workflow with falcon process
> > > > > > >
> > > > > > >  Also got this exception
> > > > > > >
> > > > > > > 2014-11-26 17:47:42,548 ERROR CompletedActionXCommand:540
-
> > > > > > > SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[]
> APP[-]
> > > > > > > JOB[0000099-141126162609456-oozie-oozi-W]
> > > > > > > ACTION[0000099-141126162609456-oozie-oozi-W@user-action]
> > > XException,
> > > > > > > org.apache.oozie.command.CommandException: E0800: Action
it is
> > not
> > > > > > > running its in [PREP] state, action
> > > > > > > [0000099-141126162609456-oozie-oozi-W@user-action]
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > _____________________________________________________________
> > > > > > The information contained in this communication is intended
> solely
> > > for
> > > > the
> > > > > > use of the individual or entity to whom it is addressed and
> others
> > > > > > authorized to receive it. It may contain confidential or legally
> > > > privileged
> > > > > > information. If you are not the intended recipient you are hereby
> > > > notified
> > > > > > that any disclosure, copying, distribution or taking any action
> in
> > > > reliance
> > > > > > on the contents of this information is strictly prohibited and
> may
> > be
> > > > > > unlawful. If you have received this communication in error,
> please
> > > > notify
> > > > > > us immediately by responding to this email and then delete it
> from
> > > your
> > > > > > system. The firm is neither liable for the proper and complete
> > > > transmission
> > > > > > of the information contained in this communication nor for any
> > delay
> > > > in its
> > > > > > receipt.
> > > > > >
> > > >
> > > >
> > >
> >
> > --
> > _____________________________________________________________
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
> >
>

-- 
_____________________________________________________________
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message