falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Tiwari <siddharth.tiw...@live.com>
Subject Re: Installation procedure for Faclon for Apache hadoop
Date Thu, 20 Aug 2015 06:55:15 GMT
Thanks Pallavi, i still had the issue, will look into the jira and patch up. 

Sent from my iPhone

> On Aug 19, 2015, at 11:53 PM, Pallavi Rao <pallavi.rao@inmobi.com> wrote:
> 
> Siddharth,
> Not sure if you were able to resolve the issue, we realized there was issue with EL expression
resolution on Apache Oozie installation. Hence, the issue you noticed. Here is the JIRA with
details and patch.
> 
> Hope this helps,
> Pallavi
> 
>> On Thu, Aug 13, 2015 at 9:12 AM, Pallavi Rao <pallavi.rao@inmobi.com> wrote:
>> Siddharth,
>> The only reason that error could occur is when the EL extension of Falcon is not
getting picked up by Oozie. Can you double check the following:
>> 1. oozie-site.xml has the extensions. You must have the following lines:
>> dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo and 
>> 
>> dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn,
>> 
>> 2. You must have the following file in the oozie installation path :
>> 
>> oozie-server/webapps/oozie/WEB-INF/lib/falcon-oozie-el-extension-0.7-SNAPSHOT.jar.
It must have read all permission.
>> 
>> 3. Hope you have restarted Oozie.
>> 
>> If you are seeing errors even after ensuring the above. Please mail across the Oozie
logs, please.
>> 
>> Regards,
>> 
>> Pallavi
>> 
>> 
>>> On Sun, Aug 9, 2015 at 10:52 AM, Siddharth Tiwari <siddharth.tiwari@live.com>
wrote:
>>> Hi Pallavi
>>> I made the changes you suggested, I am still getting the following error
>>> 
>>> Caused by: org.apache.falcon.FalconException: E1004 : E1004: Expression language
evaluation error, Unable to evaluate :${dataIn('input', 'null')}:
>>>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223)
>>>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171)
>>>         at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93)
>>>         ... 60 more
>>> Caused by: E1004 : E1004: Expression language evaluation error, Unable to evaluate
:${dataIn('input', 'null')}:
>>>         at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
>>>         at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:625)
>>>         at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:595)
>>>         at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
>>>         at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735)
>>>         at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
>>>         at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
>>>         at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
>>>         at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
>>>         at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
>>>         at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
>>> 
>>> The process I am trying to schedule is the one below:-
>>> 
>>> 
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!--
>>>  Hourly process, cleanses raw data
>>>  -->
>>> <process name="cleanseEmailProcess" xmlns="uri:falcon:process:0.1">
>>> 
>>>     <tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup</tags>
>>> 
>>> 
>>>     <clusters>
>>>         <cluster name="primaryCluster">
>>>             <validity start="2014-02-28T00:00Z" end="2016-03-31T00:00Z"/>
>>>         </cluster>
>>>     </clusters>
>>> 
>>>     <parallel>1</parallel>
>>>     <order>FIFO</order>
>>>     <frequency>hours(1)</frequency>
>>> 
>>>     <inputs>
>>>         <input name="input" feed="rawEmailFeed" start="now(0,0)" end="now(0,0)"
/>
>>>     </inputs>
>>> 
>>>     <outputs>
>>>         <output name="output" feed="cleansedEmailFeed" instance="now(0,0)"
/>
>>>     </outputs>
>>> 
>>>     <workflow name="emailCleanseWorkflow" version="5.0"
>>>     engine="pig" path="/user/ambari-qa/falcon/demo/apps/pig/id.pig" />
>>> 
>>>     <retry policy="periodic" delay="minutes(15)" attempts="3" />
>>>     <!-- <late-process policy="exp-backoff" delay="hours(1)">
>>>      <late-input input="input" workflow-path="/apps/clickstream/late" />
>>>      </late-process> -->
>>> 
>>> </process>
>>> 
>>> 
>>> 
>>> 
>>> *------------------------*
>>> Cheers !!!
>>> Siddharth Tiwari
>>> Have a refreshing day !!!
>>> "Every duty is holy, and devotion to duty is the highest form of worship of God.”

>>> "Maybe other people will try to limit me but I don't limit myself"
>>> 
>>> 
>>> > Date: Fri, 7 Aug 2015 14:03:50 +0530
>>> > Subject: Re: Installation procedure for Faclon for Apache hadoop
>>> > From: pallavi.rao@inmobi.com
>>> > To: dev@falcon.apache.org
>>> > CC: sriksun@hotmail.com
>>> > 
>>> > Hey Siddharth,
>>> > If you are not using Oozie distro that is packaged during falcon build,
>>> > ensure the following is Oozie changes are done before you start Falcon
>>> > server.
>>> > 
>>> > 1.
>>> > 
>>> > Copy <falcon base
>>> > dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
>>> > to
>>> > <oozie base dir>/libext.
>>> > 2.
>>> > 
>>> > Modify oozie site xml. Copy all EL related properties from <falcon
>>> > install dir>oozie/conf/oozie-site.xml to <oozie base
>>> > dir>/conf/oozie-site.xml
>>> > 3.
>>> > 
>>> > Restart oozie :
>>> > 
>>> > 
>>> > -
>>> > 
>>> > sudo -u oozie ./bin/oozie-start.sh
>>> > -
>>> > 
>>> > sudo -u oozie bin/oozie-setup.sh prepare-war
>>> > -
>>> > 
>>> > sudo -u oozie ./bin/oozie-start.sh
>>> > 
>>> > Just realized we have missed this information in our installation docs.
>>> > Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378>
for
>>> 
>>> > the same.
>>> > 
>>> > I assume that you have information on how to setup up a standalone Oozie
>>> > instance. If not, let me know, I can pass along the same to you.
>>> > 
>>> > Regards,
>>> > Pallavi
>>> > 
>>> > On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <siddharth.tiwari@live.com>
>>> > wrote:
>>> > 
>>> > > Hi Srikanth
>>> > > I am able to build it successfully in embedded mode, the issue is
>>> > > primarily interacting with oozie and setting up oozie to interact with
>>> > > Falcon, are there any particular steps to setup oozie for Falcon ?
Secondly
>>> > > is the version packed with HDP different from the one available in
the git
>>> > > ? Because I cannot see the xml forms setup on the gui.
>>> > >
>>> > > Sent from my iPhone
>>> > >
>>> > > > On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <sriksun@hotmail.com>
>>> > > wrote:
>>> > > >
>>> > > > HI Siddharth,
>>> > > > It should be fairly straight forward to use Falcon with hadoop
2.5.0 and
>>> > > above. Standard installation instructions should suffice. Is there
a
>>> > > particular step where you are stuck in getting it setup ?
>>> > > >
>>> > > > Regards
>>> > > > Srikanth Sundarrajan
>>> > > >
>>> > > >> From: siddharth.tiwari@live.com
>>> > > >> To: dev@falcon.apache.org
>>> > > >> Subject: Installation procedure for Faclon for Apache hadoop
>>> > > >> Date: Fri, 7 Aug 2015 07:13:52 +0000
>>> > > >>
>>> > > >> Hi Team,is there any documentation available for installing
and
>>> > > configuring falcon for stock vanila Apache hadoop ? It fairly easier
fro
>>> > > HDP using Ambari, but the procedure is not very clear for installing
it
>>> > > with Stock Apache hadoop. Any help in this regard will be highly
>>> > > appreciated. Thank you in advanced.
>>> > > >>
>>> > > >> *------------------------*
>>> > > >>
>>> > > >> Cheers !!!
>>> > > >>
>>> > > >> Siddharth Tiwari
>>> > > >>
>>> > > >> Have a refreshing day !!!
>>> > > >> "Every duty is holy, and devotion to duty is the highest form
of
>>> > > worship of God.”
>>> > > >>
>>> > > >> "Maybe other people will try to limit me but I don't limit
myself"
>>> > > >
>>> > >
>>> > 
>>> > -- 
>>> > _____________________________________________________________
>>> > The information contained in this communication is intended solely for the

>>> > use of the individual or entity to whom it is addressed and others 
>>> > authorized to receive it. It may contain confidential or legally privileged

>>> > information. If you are not the intended recipient you are hereby notified

>>> > that any disclosure, copying, distribution or taking any action in reliance

>>> > on the contents of this information is strictly prohibited and may be 
>>> > unlawful. If you have received this communication in error, please notify

>>> > us immediately by responding to this email and then delete it from your

>>> > system. The firm is neither liable for the proper and complete transmission

>>> > of the information contained in this communication nor for any delay in
its 
>>> > receipt.
> 
> 
> _____________________________________________________________
> The information contained in this communication is intended solely for the use of the
individual or entity to whom it is addressed and others authorized to receive it. It may contain
confidential or legally privileged information. If you are not the intended recipient you
are hereby notified that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be unlawful. If you have
received this communication in error, please notify us immediately by responding to this email
and then delete it from your system. The firm is neither liable for the proper and complete
transmission of the information contained in this communication nor for any delay in its receipt.
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message