hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From john smith <js1987.sm...@gmail.com>
Subject Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Date Sun, 10 Mar 2013 11:14:36 GMT
Hi,

It clearly is  a classpath issue!  When you do a select * from tab , it
works because HIve just fetches the data from HDFS using a FetchTask and
doesn't start any MR job (It probably uses json jar in your local hive lib
directory to deserialize and limit the rows to 5 and hence not raising any
error).

However the error you are facing in the 2nd query involving an MR job is
because mapper (on some remote machine which doesn't have json jar in its
class path) is unable to locate  your json jar ! Did u try doing what Dean
suggested?

Thanks

On Sun, Mar 10, 2013 at 1:49 PM, Sai Sai <saigraph@yahoo.in> wrote:

> Just wondering if anyone has any suggestions:
>
> This executes successfully:
>
> hive> select * from twitter limit 5;
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5; // I have given the exception
> info below:
>
> Here is the output of this:
>
> hive> select * from twitter limit 5;
> OK
>
> tweet_id    created_at    text    user_id    user_screen_name    user_lang
> 122106088022745088    Fri Oct 07 00:28:54 +0000 2011    wkwkw -_- ayo saja
> mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas
> galih jg kalo dia mau.. "@Dindnf: doremifas    124735434    Dindnf    en
> 122106088018558976    Fri Oct 07 00:28:54 +0000 2011    @egg486 특별히
> 준비했습니다!    252828803    CocaCola_Korea    ko
> 122106088026939392    Fri Oct 07 00:28:54 +0000 2011    My offer of free
> gobbies for all if @amityaffliction play Blair snitch project still
> stands.    168590073    SarahYoungBlood    en
> 122106088035328001    Fri Oct 07 00:28:54 +0000 2011    the girl nxt to me
> in the lib got her headphones in dancing and singing loud af like she the
> only one here haha    267296295    MONEYyDREAMS_    en
> 122106088005971968    Fri Oct 07 00:28:54 +0000 2011    @KUnYoong_B2UTY
> Bị lsao đấy    269182160    b2st_b2utyhp    en
> Time taken: 0.154 seconds
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5;
>
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201303050432_0094, Tracking URL =
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job
> -kill job_201303050432_0094
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2013-03-10 00:14:44,509 Stage-1 map = 0%,  reduce = 0%
> 2013-03-10 00:15:14,613 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201303050432_0094 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Examining task ID: task_201303050432_0094_m_000002 (and more) from job
> job_201303050432_0094
>
> Task with the most failures(4):
> -----
> Task ID:
>   task_201303050432_0094_m_000000
>
> URL:
>
> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: Error in configuring object
>     at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>     at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>     at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:416)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>     at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>     ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
>     at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>     at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>     at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>     ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>     ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
>     at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
>     ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
>     at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
>     at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
>     ... 22 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
>
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:264)
>     at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
>     at
> org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
>     at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
>     ... 23 more
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
> Thanks
> Sai
>   ------------------------------
> *From:* Dean Wampler <dean.wampler@thinkbiganalytics.com>
> *To:* user@hive.apache.org; Sai Sai <saigraph@yahoo.in>
> *Sent:* Friday, 8 March 2013 5:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> Unfortunately, you have to also add the json jars to Hive's class path
> before it starts, e.g.,
>
> env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>
> Use the appropriate path to your lib directory.
>
> On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <saigraph@yahoo.in> wrote:
>
> I have added the jar files successfully like this:
>
>
> hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
>                Added lib/hive-json-serde-0.3.jar to class path
>                Added resource: lib/hive-json-serde-0.3.jar
>
>
> hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
>                Added lib/json-path-0.5.4.jar to class path
>                Added resource: lib/json-path-0.5.4.jar
>
>
> hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
>                Added lib/json-smart-1.0.6.3.jar to class path
>                Added resource: lib/json-smart-1.0.6.3.jar
>
>
> After this i am getting this error:
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at
> STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang
> STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde"
> WITH SERDEPROPERTIES (
> "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.
> user.id","user_screen_name"="$.user.screen_name",
> "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
> java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
>     at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown
> Source)
>     at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
>     at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
>     at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
>     at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
>     at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
>     at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
>     at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
>     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
>     at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
>     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
>     at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
>     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>     ... 23 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
>
> Any help would be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
>
>

Mime
View raw message