hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5518) ADD JAR should add entries to local classpath
Date Fri, 11 Oct 2013 17:50:43 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792871#comment-13792871
] 

Nick Dimiduk commented on HIVE-5518:
------------------------------------

A related question: what's the precise behavior of using {{ADD JAR}} ? Does it copy the file
from the client local machine up to the job's distributed cache and include it on the task
classpaths?

> ADD JAR should add entries to local classpath
> ---------------------------------------------
>
>                 Key: HIVE-5518
>                 URL: https://issues.apache.org/jira/browse/HIVE-5518
>             Project: Hive
>          Issue Type: Bug
>          Components: CLI
>    Affects Versions: 0.12.0
>            Reporter: Nick Dimiduk
>
> Jars referenced in {{ADD JAR}} statements are not made available on the immediate classpath.
That means they're useless for scripts which need to initialize external output formats for
job submission (ie, hbase storage handler). Is this expected behavior?
> For example, the table 'pagecounts_hbase' is an hbase table defined using the HBaseStorageHandler
> {noformat}
> $ cat foo.hql
> ADD FILE /etc/hbase/conf/hbase-site.xml;
> ADD JAR /usr/lib/hbase/lib/hbase-common-0.96.0.2.0.6.0-68-hadoop2.jar;
> ADD JAR /usr/lib/hbase/lib/hbase-server-0.96.0.2.0.6.0-68-hadoop2.jar;
> ADD JAR /usr/lib/hbase/lib/hbase-client-0.96.0.2.0.6.0-68-hadoop2.jar;
> ADD JAR /usr/lib/hbase/lib/hbase-protocol-0.96.0.2.0.6.0-68-hadoop2.jar;
> FROM pgc INSERT INTO TABLE pagecounts_hbase SELECT pgc.* WHERE rowkey LIKE 'en/q%' LIMIT
10;
> $ hive -f foo.hql
> ...
> Added resource: /etc/hbase/conf/hbase-site.xml
> Added /usr/lib/hbase/lib/hbase-common-0.96.0.2.0.6.0-68-hadoop2.jar to class path
> Added resource: /usr/lib/hbase/lib/hbase-common-0.96.0.2.0.6.0-68-hadoop2.jar
> ...
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase
                                                                               [29/1858]
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:266)
>         at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:305)
>         at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:98)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:989)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:892)
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.<init>(BaseSemanticAnalyzer.java:730)
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.<init>(BaseSemanticAnalyzer.java:707)
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1196)
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1053)
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8342)
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>         at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>         at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
>         at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
>         at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:446)
>         at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:456)
>         at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:737)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>         ... 48 more
> {noformat}
> The ADDed jar hbase-server.jar contains the missing class:
> {noformat}
> $ unzip -l /usr/lib/hbase/lib/hbase-server-0.96.0.2.0.6.0-68-hadoop2.jar | grep TableInputFormatBase
>      5363  10-09-2013 19:45   org/apache/hadoop/hbase/mapred/TableInputFormatBase.class
>      7460  10-09-2013 19:45   org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.class
>      8803  10-09-2013 19:45   org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.class
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message