falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Greene <m...@markgreene.com>
Subject Re: Falcon Process fails with Pig HCatalog Usage
Date Fri, 22 Jan 2016 22:00:39 GMT
Sowmya,

I modified the *workflow.xml* generated by Falcon to include hive in
the oozie.action.sharelib.for.pig, and the Pig action succeeded!

What I'm struggling now to understand is why Falcon is not using the
property as defined in my oozie-site.xml, which is correct.

Mark

On Fri, Jan 22, 2016 at 2:35 PM, Sowmya Ramesh <sramesh@hortonworks.com>
wrote:

> Mark - I looked at pig-action.xml and it has
>
> <property>
>                 <name>oozie.action.sharelib.for.pig</name>
>                 <value>pig,hcatalog</value>
>             </property>
>
> Looks like even hive is required in the sharelib. Can you attach the pig
> script and process xml too? Also can you try to modify the workflow
> generated by Falcon to have hive in ³oozie.action.sharelib.for.pig² and
> try rerun the workflow and see if it succeeds?
>
> Thanks!
>
>
> On 1/22/16, 9:02 AM, "Mark Greene" <mark@markgreene.com> wrote:
>
> >Hi Dev List,
> >
> >I have tracked the issue to the *oozie.action.sharelib.for.pig* property
> >of
> >the Falcon-generated workflow not reflecting the
> >oozie.action.sharelib.for.pig
> >of my oozie-site.xml.
> >
> >From the workflow.xml generated by Falcon:
> >                <property>
> >                    <name>oozie.action.sharelib.for.pig</name>
> >                    <value>pig,hcatalog</value>
> >                </property>
> >
> >From the oozie-site.xml of the cluster:
> >    <property>
> >      <name>oozie.action.sharelib.for.pig</name>
> >      <value>hive,pig,hcatalog</value>
> >    </property>
> >
> >JIRA has an Issue logged but indicates it is resolved with Oozie versions
> >4+. My stack is running Oozie 4.2.0.
> >https://issues.apache.org/jira/browse/FALCON-243
> >
> >Any advice is appreciated.
> >
> >Mark
> >
> >
> >On Fri, Jan 22, 2016 at 10:16 AM, Mark Greene <mark@markgreene.com>
> wrote:
> >
> >> Hi Dev List,
> >>
> >> I'm trying to best understand how to troubleshoot this error (is it
> >>Oozie,
> >> or Falcon induced?).
> >>
> >> I have a Pig script that I am using as the workflow for my Falcon
> >>process.
> >> The pig script uses HCatalogStorer to write to a HCatalog URI that is
> >>the
> >> output feed defined in my Falcon Process Entity. The Pig action in the
> >> resulting Ooozie Workflow generated by Falcon fails with the attached
> >>stack
> >> trace. The root is that it is *missing a class definitions of
> >> org/apache/hadoop/hive/shims/ShimLoader.*
> >>
> >> Running the script manually using pig -x tex -useHCatalog <all the
> >>-params
> >> passed by Oozie> <path to pig script> results in a successful execution.
> >> It's only once this is called as a Pig activity in the Falcon-generated
> >> Oozie workflow that the missing class definitions manifests.
> >>
> >> I am running the following stack:
> >>
> >> HDP-2.3.2.0-2950
> >> Pig 0.15.0.2.3
> >> Hive 1.2.1.2.3
> >> Oozie 4.2.0.2.3
> >> Falcon 0.6.1.2.3
> >>
> >> --
> >>
> >> Mark Greene
> >> *E:* Mark@MarkGreene.com
> >>
> >
> >
> >
> >--
> >
> >Mark Greene
> >*E:* Mark@MarkGreene.com
> >*T: *+1 512 663 0445
>
>


-- 

Mark Greene
*E:* Mark@MarkGreene.com
*T: *+1 512 663 0445

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message