hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18803) Mapreduce job get failed caused by NoClassDefFoundError: org/apache/commons/lang3/ArrayUtils
Date Thu, 14 Sep 2017 18:50:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166808#comment-16166808
] 

Yi Liang commented on HBASE-18803:
----------------------------------

Hi [~Apache9]
  I just debug the code and found the issue, it seems that for mapreduce job, it will load
both its own hadoop class&lib path and also hbase class&lib path. I think that is
the conflicts come from. See debug info below.

{quote}
*I write a tiny test to show which curator jar was used in mr job, Below is the real curator
jar used in mapreduce job*
Error: java.io.IOException: PATH IS file:/usr/local/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar/

*Below is all the jars added into mapreduce job, you cam see both 2.7.1 and 2.12 are loaded*
file:/usr/local/hbase/lib/jackson-mapper-asl-1.9.13.jar,file:/usr/local/hbase/lib/hbase-server-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/hbase-common-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/hbase-metrics-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/htrace-core-3.2.0-incubating.jar,file:/usr/local/hbase/lib/hbase-hadoop2-compat-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/zookeeper-3.4.10.jar,file:/usr/local/hbase/lib/hbase-hadoop-compat-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/protobuf-java-2.5.0.jar,file:/usr/local/hbase/lib/hbase-protocol-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/hbase-prefix-tree-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/hbase-shaded-protobuf-1.0.1.jar,file:/usr/local/hbase/lib/hbase-protocol-shaded-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/curator-framework-2.12.0.jar,file:/usr/local/hbase/lib/hbase-mapreduce-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/hbase-shaded-miscellaneous-1.0.1.jar,file:/usr/local/hbase/lib/hadoop-mapreduce-client-core-2.7.1.jar,file:/usr/local/hbase/lib/hbase-metrics-api-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/hbase-shaded-netty-1.0.1.jar,file:/usr/local/hbase/lib/metrics-core-3.2.1.jar,file:/usr/local/hbase/lib/hbase-client-3.0.0-SNAPSHOT.jar,file:/usr/local/hbase/lib/commons-lang3-3.6.jar,file:/usr/local/hbase/lib/hadoop-common-2.7.1.jar
{quote}


If we use shaded jar for curator, maybe this can be solved.


BTW [~stack] could you help to commit this one, so we can open another jira to fix curator
issues.. Thanks

> Mapreduce job get failed caused by NoClassDefFoundError: org/apache/commons/lang3/ArrayUtils
> --------------------------------------------------------------------------------------------
>
>                 Key: HBASE-18803
>                 URL: https://issues.apache.org/jira/browse/HBASE-18803
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0-alpha-2
>            Reporter: Yi Liang
>            Assignee: Yi Liang
>             Fix For: 2.0.0
>
>         Attachments: HBASE-18803-v1-master.patch
>
>
> 2017-09-12 17:44:13,020 INFO  [main] mapreduce.Job: Task Id : attempt_1505170797673_0010_m_000000_0,
Status : FAILED
> Error: java.io.IOException: java.lang.reflect.InvocationTargetException
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:221)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
> 	at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:311)
> 	at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:264)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
> 	... 11 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang3/ArrayUtils
> 	at org.apache.hadoop.hbase.HConstants.<clinit>(HConstants.java:1144)
> 	at org.apache.hadoop.hbase.client.ConnectionUtils.getPauseTime(ConnectionUtils.java:84)
> 	at org.apache.hadoop.hbase.client.AsyncProcess.<init>(AsyncProcess.java:209)
> 	at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:266)
> 	... 16 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message