falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Agarwal <gzagar...@gmail.com>
Subject Re: Need help for the problem that i faced
Date Fri, 28 Nov 2014 04:55:26 GMT
thanks issue is resolved.

On 11/27/14, Shwetha GS <shwetha.gs@inmobi.com> wrote:
> Try the java action in oozie:
> http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a3.2.7_Java_Action
>
> On Thu, Nov 27, 2014 at 11:49 AM, Gaurav Agarwal <gzagarwal@gmail.com>
> wrote:
>
>> Shwetha
>>
>> Apologise to trouble everyone but i am new to this falcon. Do i need to
>> check the logs in JobBrowser?
>>
>> there is stderror
>>
>> Log Length: 188
>> Error: Could not find or load main class com.jpmc.cib.is.bigdata.
>> falcon.SemsStaticDataProvider
>> Failing Oozie Launcher, Main class
>> [org.apache.oozie.action.hadoop.ShellMain],
>> exit code [1]
>>
>>
>>
>>
>>
>> On Thu, Nov 27, 2014 at 11:43 AM, Shwetha GS <shwetha.gs@inmobi.com>
>> wrote:
>>
>> > oozie launches a hadoop job which executes the shell script. Check the
>> > launcher job logs
>> >
>> > On Thu, Nov 27, 2014 at 11:39 AM, Gaurav Agarwal <gzagarwal@gmail.com>
>> > wrote:
>> >
>> > > Srikanth
>> > >
>> > > What do we mean by that in oozie context?
>> > >
>> > > Thanks
>> > >
>> > > On Thu, Nov 27, 2014 at 11:36 AM, Srikanth Sundarrajan <
>> > > sriksun@hotmail.com>
>> > > wrote:
>> > >
>> > > > Looks like the script ingest.sh is returning non-zero exit code.
>> > > >
>> > > > Regards
>> > > > Srikanth Sundarrajan
>> > > >
>> > > > > Date: Thu, 27 Nov 2014 11:31:30 +0530
>> > > > > Subject: Re: Need help for the problem that i faced
>> > > > > From: gzagarwal@gmail.com
>> > > > > To: dev@falcon.incubator.apache.org
>> > > > >
>> > > > > These are the log
>> > > > >
>> > > > > 2014-11-27 05:37:02,726 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@:start:] Start action
>> > > > > [0000129-141126162609456-oozie-oozi-W@:start:] with user-retry
>> > state :
>> > > > > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>> > > > > 2014-11-27 05:37:02,727 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@:start:]
>> > > > > [***0000129-141126162609456-oozie-oozi-W@:start:***]Action
>> > status=DONE
>> > > > > 2014-11-27 05:37:02,727 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@:start:]
>> > > > > [***0000129-141126162609456-oozie-oozi-W@:start:***]Action
>> > > > > updated
>> > in
>> > > > DB!
>> > > > > 2014-11-27 05:37:03,151 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] Start
>> action
>> > > > > [0000129-141126162609456-oozie-oozi-W@shell-node] with user-retry
>> > > state
>> > > > :
>> > > > > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>> > > > > 2014-11-27 05:37:05,377 INFO ShellActionExecutor:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] checking
>> > > action,
>> > > > > hadoop job ID [job_1417019189193_0117] status [RUNNING]
>> > > > > 2014-11-27 05:37:05,378 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node]
>> > > > > [***0000129-141126162609456-oozie-oozi-W@shell-node***]Action
>> > > > > status=RUNNING
>> > > > > 2014-11-27 05:37:05,379 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node]
>> > > > > [***0000129-141126162609456-oozie-oozi-W@shell-node***]Action
>> > updated
>> > > in
>> > > > > DB!
>> > > > > 2014-11-27 05:37:49,260 INFO CallbackServlet:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] callback
>> for
>> > > > action
>> > > > > [0000129-141126162609456-oozie-oozi-W@shell-node]
>> > > > > 2014-11-27 05:37:50,753 INFO ShellActionExecutor:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] action
>> > > > completed,
>> > > > > external ID [job_1417019189193_0117]
>> > > > > 2014-11-27 05:37:50,761 WARN ShellActionExecutor:546 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] Launcher
>> > > ERROR,
>> > > > > reason: Main class [org.apache.oozie.action.hadoop.ShellMain],
>> > > > > exit
>> > > code
>> > > > > [1]
>> > > > > 2014-11-27 05:37:50,865 INFO ActionEndXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@shell-node] ERROR
is
>> > > > considered
>> > > > > as FAILED for SLA
>> > > > > 2014-11-27 05:37:50,941 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@fail] Start action
>> > > > > [0000129-141126162609456-oozie-oozi-W@fail] with user-retry state
>> :
>> > > > > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>> > > > > 2014-11-27 05:37:50,942 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@fail]
>> > > > > [***0000129-141126162609456-oozie-oozi-W@fail***]Action
>> status=DONE
>> > > > > 2014-11-27 05:37:50,942 INFO ActionStartXCommand:543 - SERVER[
>> > > > > sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[]
>> > > APP[shell-wf]
>> > > > > JOB[0000129-141126162609456-oozie-oozi-W]
>> > > > > ACTION[0000129-141126162609456-oozie-oozi-W@fail]
>> > > > > [***0000129-141126162609456-oozie-oozi-W@fail***]Action updated
>> > > > > in
>> > DB!
>> > > > >
>> > > > > On Thu, Nov 27, 2014 at 10:25 AM, Shwetha GS <
>> shwetha.gs@inmobi.com>
>> > > > wrote:
>> > > > >
>> > > > > >   <exec>ingest.sh</exec>
>> > > > > > Try using absolute path.
>> > > > > >
>> > > > > > None of the logs imply any error. Can you attach all the
oozie
>> logs
>> > > > for the
>> > > > > > workflow?
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Nov 27, 2014 at 7:50 AM, Gaurav Agarwal <
>> > gzagarwal@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hello I am running apache falcon and oozie with shell
script
>> > > > > > > to
>> > run
>> > > > the
>> > > > > > > workflow.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > getting this error
>> > > > > > >
>> > > > > > > *ell-node] Launcher ERROR, reason: Main class
>> > > > > > > [org.apache.oozie.action.hadoop.ShellMain], exit code
[1]*
>> > > > > > > *2014-11-26 17:24:06,645  INFO ActionEndXCommand:543
-
>> > > > > > > SERVER[sandbox.hortonworks.com
>> > > > > > > <http://sandbox.hortonworks.com
>> >]
>> > > > > > > USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf]
>> > > > > > > JOB[0000020-141126162609456-oozie-oozi-W]
>> > > > > > > ACTION[0000020-141126162609456-oozie-oozi-W@shell-node]
ERROR
>> is
>> > > > > > > considered
>> > > > > > > as FAILED for SLA*
>> > > > > > > *2014-11-26 17:24:06,702  INFO ActionStartXCommand:543
-
>> > > > > > > SERVER[sandbox.hortonworks.com
>> > > > > > > <http://sandbox.hortonworks.com
>> >]
>> > > > > > > USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf]
>> > > > > > > JOB[0000020-141126162609456-oozie-oozi-W]
>> > > > > > > ACTION[0000020-141126162609456-oozie-oozi-W@fail] Start
>> > > > > > > action
>> > > > > > > [0000020-141126162609456-oozie-*
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > *Workflow*
>> > > > > > >
>> > > > > > > <!--
>> > > > > > >   Licensed to the Apache Software Foundation (ASF)
under one
>> > > > > > >   or more contributor license agreements.  See the
NOTICE
>> > > > > > > file
>> > > > > > >   distributed with this work for additional information
>> > > > > > >   regarding copyright ownership.  The ASF licenses
this file
>> > > > > > >   to you under the Apache License, Version 2.0 (the
>> > > > > > >   "License"); you may not use this file except in compliance
>> > > > > > >   with the License.  You may obtain a copy of the License
at
>> > > > > > >
>> > > > > > >        http://www.apache.org/licenses/LICENSE-2.0
>> > > > > > >
>> > > > > > >   Unless required by applicable law or agreed to in
writing,
>> > > software
>> > > > > > >   distributed under the License is distributed on an
"AS IS"
>> > BASIS,
>> > > > > > >   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
>> > > > > > > express
>> or
>> > > > > > implied.
>> > > > > > >   See the License for the specific language governing
>> permissions
>> > > and
>> > > > > > >   limitations under the License.
>> > > > > > > -->
>> > > > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">
>> > > > > > >     <start to="shell-node"/>
>> > > > > > >     <action name="shell-node">
>> > > > > > >         <shell xmlns="uri:oozie:shell-action:0.2">
>> > > > > > >             <job-tracker>${jobTracker}</job-tracker>
>> > > > > > >             <name-node>${nameNode}</name-node>
>> > > > > > >             <configuration>
>> > > > > > >                 <property>
>> > > > > > >                     <name>mapred.job.queue.name</name>
>> > > > > > >                     <value>${queueName}</value>
>> > > > > > >                 </property>
>> > > > > > >             </configuration>
>> > > > > > >             <exec>ingest.sh</exec>
>> > > > > > >             <argument>${feedInstancePaths}</argument>
>> > > > > > >             <file>${wf:appPath()}/ingest.sh#ingest.sh</file>
>> > > > > > >             <!-- <file>/tmp/ingest.sh#ingest.sh</file>
-->
>> > > > > > >             <!-- <capture-output/> -->
>> > > > > > >         </shell>
>> > > > > > >         <ok to="end"/>
>> > > > > > >         <error to="fail"/>
>> > > > > > >     </action>
>> > > > > > >     <kill name="fail">
>> > > > > > >         <message>Shell action failed, error
>> > > > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>> > > > > > >     </kill>
>> > > > > > >     <end name="end"/>
>> > > > > > > </workflow-app>
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > This is the shell script
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > #!/bin/bash
>> > > > > > >
>> > > > > > >
>> > > > > > > java -cp
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> "/media/sf_ToSandbox/BigData-0.0.1-SNAPSHOT.jar:/usr/hdp/2.2.0.0-913/sqoop/lib/*:/usr/hdp/2.2.0.0-913/hadoop/lib/*"
>> > > > > > > com.bigdata.falcon.StaticDataProvider
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > These are the feed and process and cluster information
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > <feed description="Raw customer email feed"
>> > > name="FINALrawEmailFeed"
>> > > > > > >     xmlns="uri:falcon:feed:0.1">
>> > > > > > >
>> > > > > > >
>> > > >
>> > > > <tags>externalSystem=USWestEmailServers,classification=secure</tags>
>> > > > > > >     <groups>churnAnalysisDataPipeline</groups>
>> > > > > > >     <frequency>minutes(3)</frequency>
>> > > > > > >     <late-arrival cut-off="hours(4)"/>
>> > > > > > >     <clusters>
>> > > > > > >         <cluster name="SprimaryCluster" type="source">
>> > > > > > >             <validity start="2014-11-21T00:00Z"
>> > > > end="2016-03-31T00:00Z"/>
>> > > > > > >             <retention limit="days(90)" action="delete"/>
>> > > > > > >         </cluster>
>> > > > > > >     </clusters>
>> > > > > > >
>> > > > > > >     <locations>
>> > > > > > >         <location type="data"
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> path="/user/gaurav123/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
>> > > > > > >         <location type="stats" path="/none"/>
>> > > > > > >         <location type="meta" path="/none"/>
>> > > > > > >     </locations>
>> > > > > > >
>> > > > > > >     <ACL owner="ambari-qa" group="users" permission="0777"/>
>> > > > > > >     <schema location="/none" provider="none"/>
>> > > > > > > </feed>
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > *Process*
>> > > > > > >
>> > > > > > > *-----------*
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > <?xml version="1.0" encoding="UTF-8"?>
>> > > > > > > <!--
>> > > > > > >  Hourly process, ingests raw emails from all US west
email
>> > servers
>> > > > > > >  -->
>> > > > > > > <process name="IngestProcess" xmlns="uri:falcon:process:0.1">
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> <tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup,externalSystem=USWestEmailServers</tags>
>> > > > > > >
>> > > > > > >
>> > > > > > >     <clusters>
>> > > > > > >         <cluster name="SprimaryCluster">
>> > > > > > >             <validity start="2014-11-24T09:35Z"
>> > > > end="2014-11-25T10:00Z"/>
>> > > > > > >         </cluster>
>> > > > > > >     </clusters>
>> > > > > > >
>> > > > > > >     <parallel>1</parallel>
>> > > > > > >     <order>FIFO</order>
>> > > > > > >     <frequency>minutes(3)</frequency>
>> > > > > > >
>> > > > > > >     <outputs>
>> > > > > > >         <output name="output" feed="FINALrawEmailFeed"
>> > > > > > instance="now(0,0)"
>> > > > > > > />
>> > > > > > >     </outputs>
>> > > > > > >
>> > > > > > >     <workflow name="IngestWorkflow" version="2.0.0"
>> > > > > > >     engine="oozie"
>> > > path="/user/gaurav123/falcon/demo/apps/ingest/fs"
>> > > > />
>> > > > > > >
>> > > > > > >     <retry policy="periodic" delay="minutes(15)"
attempts="3"
>> />
>> > > > > > >
>> > > > > > > </process>
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Can you please let me know what is the issue , I have
tried
>> since
>> > > > long
>> > > > > > two
>> > > > > > > days? but unable to run the workflow with falcon process
>> > > > > > >
>> > > > > > >  Also got this exception
>> > > > > > >
>> > > > > > > 2014-11-26 17:47:42,548 ERROR CompletedActionXCommand:540
-
>> > > > > > > SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[]
>> APP[-]
>> > > > > > > JOB[0000099-141126162609456-oozie-oozi-W]
>> > > > > > > ACTION[0000099-141126162609456-oozie-oozi-W@user-action]
>> > > XException,
>> > > > > > > org.apache.oozie.command.CommandException: E0800: Action
it
>> > > > > > > is
>> > not
>> > > > > > > running its in [PREP] state, action
>> > > > > > > [0000099-141126162609456-oozie-oozi-W@user-action]
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > _____________________________________________________________
>> > > > > > The information contained in this communication is intended
>> solely
>> > > for
>> > > > the
>> > > > > > use of the individual or entity to whom it is addressed
and
>> others
>> > > > > > authorized to receive it. It may contain confidential or
>> > > > > > legally
>> > > > privileged
>> > > > > > information. If you are not the intended recipient you are
>> > > > > > hereby
>> > > > notified
>> > > > > > that any disclosure, copying, distribution or taking any
action
>> in
>> > > > reliance
>> > > > > > on the contents of this information is strictly prohibited
and
>> may
>> > be
>> > > > > > unlawful. If you have received this communication in error,
>> please
>> > > > notify
>> > > > > > us immediately by responding to this email and then delete
it
>> from
>> > > your
>> > > > > > system. The firm is neither liable for the proper and complete
>> > > > transmission
>> > > > > > of the information contained in this communication nor for
any
>> > delay
>> > > > in its
>> > > > > > receipt.
>> > > > > >
>> > > >
>> > > >
>> > >
>> >
>> > --
>> > _____________________________________________________________
>> > The information contained in this communication is intended solely for
>> the
>> > use of the individual or entity to whom it is addressed and others
>> > authorized to receive it. It may contain confidential or legally
>> privileged
>> > information. If you are not the intended recipient you are hereby
>> notified
>> > that any disclosure, copying, distribution or taking any action in
>> reliance
>> > on the contents of this information is strictly prohibited and may be
>> > unlawful. If you have received this communication in error, please
>> > notify
>> > us immediately by responding to this email and then delete it from your
>> > system. The firm is neither liable for the proper and complete
>> transmission
>> > of the information contained in this communication nor for any delay in
>> its
>> > receipt.
>> >
>>
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
>
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
>
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
>
> of the information contained in this communication nor for any delay in its
>
> receipt.
>

Mime
View raw message