incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From agateaaa <agate...@gmail.com>
Subject Re: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader
Date Sat, 25 Aug 2012 03:03:45 GMT
Look at http://incubator.apache.org/hcatalog/docs/r0.4.0/loadstore.htmlread
section "Running Pig with HCatalog"

export HADOOP_HOME=<path_to_hadoop_install>

export HCAT_HOME=<path_to_hcat_install>

export
PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/lib/hive-metastore-0.9.0.jar:
$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar:
$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar


export PIG_OPTS=-Dhive.metastore.uris=thrift://<hostname>:<port>

/* invoke pig */
<path_to_pig_install>/bin/pig
-Dpig.additional.jars=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:
$HIVE_HOME/lib/hive-metastore-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:
$HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/lib/slf4j-api-1.6.1.jar
<script.pig>


Essentially  you need to set PIG_CLASSPATH  env variable as well as set
-Dpig.additional.jars pointing to same jar files
-Dpig,additional.jars=$PIG_CLASSPATH

Here is the relevant (and fairly long) threadin the mailing list which will
give you an idea on why you need both PIG_CLASSPATH as well as set
'pig,additional.jars'
http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201206.mbox/%3CCANSvDjp1m=nuzc=T+Oi1OJmnjkc_SsycTAvn0cJJ16s=BrDWpg@mail.gmail.com%3E


Note: I am not using HDP but hopefully above would shed some light on your
problem

-Agatea

On Fri, Aug 24, 2012 at 9:58 AM, Petr Babkin <pbabkin@revelytix.com> wrote:

> Thank you, Mr. Gates!
>
> Now pig doesn't complain about hive-site.xml anymore, however it gets me
> the same error:
> *ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal
> error. com/facebook/fb303/FacebookService$Iface*
> The log file says:
> java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>         at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         at
> org.apache.hcatalog.pig.PigHCatUtil.createHiveMetaClient(PigHCatUtil.java:114)
>         at
> org.apache.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:150)
>         at
> org.apache.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:151)
>         at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
>         at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
>         at
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:100)
>         at
> org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
>         at
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>         at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
>         at
> org.apache.pig.newplan.logical.visitor.CastLineageSetter.<init>(CastLineageSetter.java:57)
>         at org.apache.pig.PigServer$Graph.compile(PigServer.java:1679)
>         at
> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1610)
>         at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>         at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>         at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>         at org.apache.pig.Main.run(Main.java:495)
>         at org.apache.pig.Main.main(Main.java:111)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException:
> com.facebook.fb303.FacebookService$Iface
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         ... 38 more
>
> It's quite surprising how I'm running into these issue having launched HDP
> from Hortonworks' pre-configured VM. I just can't imagine a place where I
> could have messed up with the configuration.
>
> Regards,
> Petr Babkin.
>
> On Fri, Aug 24, 2012 at 12:44 PM, Alan Gates <gates@hortonworks.com>wrote:
>
>> Also your PIG_CLASSPATH has to point to the directory that hive-site.xml
>> is in, not to the hive-site.xml file itself.
>>
>> Alan.
>>
>> On Aug 24, 2012, at 9:10 AM, Petr Babkin wrote:
>>
>> > Thanks, Aniket and Vandana,
>> >
>> > after adding the jars to PIG_CLASSPATH, I'm getting:
>> >
>> > WARN  org.apache.hadoop.hive.conf.HiveConf - hive-site.xml not found on
>> CLASSPATH
>> > ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal
>> error. com/facebook/fb303/FacebookService$Iface
>> >
>> > though my PIG_CLASSPATH has /etc/hive/conf.dist/hive-site.xml added.
>> >
>> > Vandana, could you please be more specific on what HCatalog
>> configuration needs to be done?
>> >
>> > Many thanks!
>> >
>> > Regards,
>> > Petr Babkin
>> >
>> > On Thu, Aug 23, 2012 at 7:02 PM, Aniket Mokashi <aniket486@gmail.com>
>> wrote:
>> > 1. Make sure HCatalog jar is in the class path (named hcatalog*.jar)
>> > 2. Make sure your pig is running with
>> -Dudf.import.list=org.apache.hcatalog.pig. This is how pig finds the
>> package prefix for HCatLoader
>> >
>> > Workaround for 2 is- in your pig script- replace HCatLoader by
>> org.apache.hcatalog.pig.HCatLoader.
>> >
>> > I assume HDP is doing all that already. So, I suspect something else is
>> responsible here.
>> >
>> > Thanks,
>> > Aniket
>> >
>> > On Thu, Aug 23, 2012 at 3:47 PM, Vandana Ayyalasomayajula <
>> avandana@yahoo-inc.com> wrote:
>> > Hi Petr,
>> >
>> > I am thinking that this issue is related to class path. You need to
>> have the HCatalog, hive jars in your PIG_CLASSPATH env variable.
>> >
>> > From a previous post in the user list, the following this are required
>> in the class path.
>> >
>> > Minimum set-
>> > /share/hcatalog/hcatalog*.jar:
>> > /share/hcatalog/lib/hive-metastore*.jar:
>> > /share/hcatalog/lib/thrift-fb303-*.jar:
>> > /share/hcatalog/lib/libthrift*.jar:
>> > /share/hcatalog/lib/hive-exec-*.jar
>> > +HCatalog configuration (hive-site et al)…
>> >
>> > Thanks
>> > Vandana
>> >
>> >
>> > On Aug 23, 2012, at 12:49 PM, Petr Babkin wrote:
>> >
>> >> I am getting the following error when trying to load the table from
>> HCatalog into pig:
>> >>
>> >> ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve
>> org.apache.hcatalog.pig.HCatLoader using imports: [,
>> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>> >>
>> >> This is all running on HDP, so I am surprised why the resolution
>> fails. Googling around this issue didn't bring any result.
>> >>
>> >> Thanks.
>> >>
>> >> --
>> >> Regards,
>> >> Petr Babkin
>> >> Ontologist, Revelytix Inc.
>> >>
>> >
>> > Vandana Ayyalasomayajula
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > "...:::Aniket:::... Quetzalco@tl"
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Petr Babkin
>> > Ontologist, Revelytix Inc.
>> >
>>
>>
>
>
> --
> Regards,
> Petr Babkin
> Ontologist, Revelytix Inc.
>
>

Mime
View raw message