incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Petr Babkin <pbab...@revelytix.com>
Subject Re: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader
Date Mon, 27 Aug 2012 15:15:06 GMT
Thank you,  Agatea!

On Fri, Aug 24, 2012 at 11:03 PM, agateaaa <agateaaa@gmail.com> wrote:

> Look at http://incubator.apache.org/hcatalog/docs/r0.4.0/loadstore.htmlread section "Running
Pig with HCatalog"
>
> export HADOOP_HOME=<path_to_hadoop_install>
>
> export HCAT_HOME=<path_to_hcat_install>
>
> export
> PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/lib/hive-metastore-0.9.0.jar:
> $HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar:
> $HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar
>
>
> export PIG_OPTS=-Dhive.metastore.uris=thrift://<hostname>:<port>
>
> /* invoke pig */
> <path_to_pig_install>/bin/pig
> -Dpig.additional.jars=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:
> $HIVE_HOME/lib/hive-metastore-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:
> $HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/lib/slf4j-api-1.6.1.jar
> <script.pig>
>
>
> Essentially  you need to set PIG_CLASSPATH  env variable as well as set
> -Dpig.additional.jars pointing to same jar files
> -Dpig,additional.jars=$PIG_CLASSPATH
>
> Here is the relevant (and fairly long) threadin the mailing list which
> will give you an idea on why you need both PIG_CLASSPATH as well as set
> 'pig,additional.jars'
>
> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201206.mbox/%3CCANSvDjp1m=nuzc=T+Oi1OJmnjkc_SsycTAvn0cJJ16s=BrDWpg@mail.gmail.com%3E
>
>
> Note: I am not using HDP but hopefully above would shed some light on your
> problem
>
> -Agatea
>
>
> On Fri, Aug 24, 2012 at 9:58 AM, Petr Babkin <pbabkin@revelytix.com>wrote:
>
>> Thank you, Mr. Gates!
>>
>> Now pig doesn't complain about hive-site.xml anymore, however it gets me
>> the same error:
>> *ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal
>> error. com/facebook/fb303/FacebookService$Iface*
>> The log file says:
>> java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>         at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>         at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>         at
>> org.apache.hcatalog.pig.PigHCatUtil.createHiveMetaClient(PigHCatUtil.java:114)
>>         at
>> org.apache.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:150)
>>         at
>> org.apache.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:151)
>>         at
>> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
>>         at
>> org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
>>         at
>> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:100)
>>         at
>> org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
>>         at
>> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>>         at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
>>         at
>> org.apache.pig.newplan.logical.visitor.CastLineageSetter.<init>(CastLineageSetter.java:57)
>>         at org.apache.pig.PigServer$Graph.compile(PigServer.java:1679)
>>         at
>> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1610)
>>         at
>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>>         at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>>         at
>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>>         at
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
>>         at
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
>>         at
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
>>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>>         at org.apache.pig.Main.run(Main.java:495)
>>         at org.apache.pig.Main.main(Main.java:111)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.lang.ClassNotFoundException:
>> com.facebook.fb303.FacebookService$Iface
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>         ... 38 more
>>
>> It's quite surprising how I'm running into these issue having launched
>> HDP from Hortonworks' pre-configured VM. I just can't imagine a place where
>> I could have messed up with the configuration.
>>
>> Regards,
>> Petr Babkin.
>>
>> On Fri, Aug 24, 2012 at 12:44 PM, Alan Gates <gates@hortonworks.com>wrote:
>>
>>> Also your PIG_CLASSPATH has to point to the directory that hive-site.xml
>>> is in, not to the hive-site.xml file itself.
>>>
>>> Alan.
>>>
>>> On Aug 24, 2012, at 9:10 AM, Petr Babkin wrote:
>>>
>>> > Thanks, Aniket and Vandana,
>>> >
>>> > after adding the jars to PIG_CLASSPATH, I'm getting:
>>> >
>>> > WARN  org.apache.hadoop.hive.conf.HiveConf - hive-site.xml not found
>>> on CLASSPATH
>>> > ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled
>>> internal error. com/facebook/fb303/FacebookService$Iface
>>> >
>>> > though my PIG_CLASSPATH has /etc/hive/conf.dist/hive-site.xml added.
>>> >
>>> > Vandana, could you please be more specific on what HCatalog
>>> configuration needs to be done?
>>> >
>>> > Many thanks!
>>> >
>>> > Regards,
>>> > Petr Babkin
>>> >
>>> > On Thu, Aug 23, 2012 at 7:02 PM, Aniket Mokashi <aniket486@gmail.com>
>>> wrote:
>>> > 1. Make sure HCatalog jar is in the class path (named hcatalog*.jar)
>>> > 2. Make sure your pig is running with
>>> -Dudf.import.list=org.apache.hcatalog.pig. This is how pig finds the
>>> package prefix for HCatLoader
>>> >
>>> > Workaround for 2 is- in your pig script- replace HCatLoader by
>>> org.apache.hcatalog.pig.HCatLoader.
>>> >
>>> > I assume HDP is doing all that already. So, I suspect something else
>>> is responsible here.
>>> >
>>> > Thanks,
>>> > Aniket
>>> >
>>> > On Thu, Aug 23, 2012 at 3:47 PM, Vandana Ayyalasomayajula <
>>> avandana@yahoo-inc.com> wrote:
>>> > Hi Petr,
>>> >
>>> > I am thinking that this issue is related to class path. You need to
>>> have the HCatalog, hive jars in your PIG_CLASSPATH env variable.
>>> >
>>> > From a previous post in the user list, the following this are required
>>> in the class path.
>>> >
>>> > Minimum set-
>>> > /share/hcatalog/hcatalog*.jar:
>>> > /share/hcatalog/lib/hive-metastore*.jar:
>>> > /share/hcatalog/lib/thrift-fb303-*.jar:
>>> > /share/hcatalog/lib/libthrift*.jar:
>>> > /share/hcatalog/lib/hive-exec-*.jar
>>> > +HCatalog configuration (hive-site et al)…
>>> >
>>> > Thanks
>>> > Vandana
>>> >
>>> >
>>> > On Aug 23, 2012, at 12:49 PM, Petr Babkin wrote:
>>> >
>>> >> I am getting the following error when trying to load the table from
>>> HCatalog into pig:
>>> >>
>>> >> ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not
>>> resolve org.apache.hcatalog.pig.HCatLoader using imports: [,
>>> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>>> >>
>>> >> This is all running on HDP, so I am surprised why the resolution
>>> fails. Googling around this issue didn't bring any result.
>>> >>
>>> >> Thanks.
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Petr Babkin
>>> >> Ontologist, Revelytix Inc.
>>> >>
>>> >
>>> > Vandana Ayyalasomayajula
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > "...:::Aniket:::... Quetzalco@tl"
>>> >
>>> >
>>> >
>>> > --
>>> > Regards,
>>> > Petr Babkin
>>> > Ontologist, Revelytix Inc.
>>> >
>>>
>>>
>>
>>
>> --
>> Regards,
>> Petr Babkin
>> Ontologist, Revelytix Inc.
>>
>>
>


-- 
Regards,
Petr Babkin
Ontologist, Revelytix Inc.

Mime
View raw message