hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jilani Shaik <jilani2...@gmail.com>
Subject Re: hbase bulk load with map reduce error
Date Wed, 15 Jun 2016 06:47:59 GMT
Please let me know what am I missing here.

I am using MapR hadoop. Please find the classpath which is showing all the
jars with versions, mean while I will get the code snippet also. I am using
bulk write approach with mapper and reducer. HBase table created with bulk
load enabled as true.

+ HADOOP_CLASSPATH='/opt/mapr/hbase/hbase-0.98.9/bin/../
conf:/opt/mapr/java/jdk1.7.0_25/lib/tools.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/..:/opt/mapr/lib/zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/activation-1.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/aopalliance-1.0.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/api-util-1.0.0-M20.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/asm-3.1.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-beanutils-1.7.0.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-cli-1.2.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-codec-1.7.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/commons-collections-3.2.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-compress-1.4.1.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-configuration-1.6.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-daemon-1.0.13.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-digester-1.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-el-1.0.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-httpclient-3.1.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-io-2.4.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-lang-2.6.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-logging-1.1.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-math-2.1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/commons-math3-3.1.1.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-net-3.1.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/findbugs-annotations-1.3.9-1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/guava-12.0.1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/guice-3.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
guice-servlet-3.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/hamcrest-core-1.3.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase
-annotations-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-checkstyle-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-client-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-common-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-common-0.98.9-mapr-1503-tests.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/hbase-examples-0.98.9-mapr-1503.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/hbase-hadoop2-compat-0.98.9-mapr-1503.jar:/opt/mapr/
hbase/hbase-0.98.9/bin/../lib/hbase-hadoop-compat-0.98.9-mapr-1503.jar:/
opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase-it-0.98.9-
mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase-
it-0.98.9-mapr-1503-tests.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase
-prefix-tree-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-protocol-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-rest-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-server-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-server-0.98.9-mapr-1503-tests.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-shell-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-testing-util-0.98.9-mapr-1503.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/hbase-thrift-0.98.9-mapr-1503.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/high-scale-lib-1.1.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/htrace-core-2.04.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/httpclient-4.2.5.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/httpcore-4.1.3.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jackson-core-asl-1.8.8.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/jackson-jaxrs-1.8.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jackson-mapper-asl-1.8.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jackson-xc-1.8.8.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/jamon-runtime-2.3.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jasper-compiler-5.5.23.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jasper-runtime-5.5.23.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/javax.inject-1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/java-xmlbuilder-0.4.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jaxb-api-2.2.2.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/jcodings-1.0.8.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/jersey-client-1.9.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jersey-core-1.8.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/jersey-guice-1.9.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/jersey-json-1.8.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/jersey-server-1.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jets3t-0.9.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jettison-1.3.1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jetty-6.1.26.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jetty-sslengine-6.1.26.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jetty-util-6.1.26.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/joni-2.1.2.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/jruby-complete-1.6.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jsch-0.1.42.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jsp-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jsp-api-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/jsr305-1.3.9.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/junit-4.11.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
leveldbjni-all-1.8.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/libthrift-0.9.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/log4j-1.2.17.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/metrics-core-2.2.0.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/netty-3.6.6.Final.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/paranamer-2.3.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/servlet-api-2.5.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/snappy-java-1.0.4.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/xmlenc-0.52.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/xz-1.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/lib/libprotodefs-4.
0.2-mapr.jar:/opt/mapr/lib/baseutils-4.0.2-mapr.jar:/opt/
mapr/lib/JPam-1.1.jar:/opt/mapr/hadoop/hadoop-2.5.1/etc/
hadoop:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/
common/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/
common/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/
hdfs:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/hdfs/lib/*:
/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/mapr/
hadoop/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/mapr/
hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/
hadoop/mapreduce/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/
share/hadoop/mapreduce/*

Thanks,
Jilani

On Sat, Jun 11, 2016 at 6:39 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Which version of hbase / Hadoop are you using ?
> (So that line number matching can be more accurate)
>
> It would be good if you can show your code snippet.
>
> Thanks
>
> On Jun 11, 2016, at 12:57 AM, Jilani Shaik <jilani2423@gmail.com> wrote:
>
> Hi,
>
> I am trying to do hbase table bulk load from data file using map reduce.
> in the process I am getting the Null pointer exception during the shuffle
> and sort. I have both mapper and reducer in the flow. where as mapper is
> ending with success and reducer not being invoked even setup.
>
> It is showing the below error in terminal where the job executed.
>
> Error: java.lang.NullPointerException
>
>         at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
>
>         at com.google.common.base.Splitter.split(Splitter.java:375)
>
>         at
> org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser.<init>(ImportTsv.java:151)
>
>         at
> org.apache.hadoop.hbase.mapreduce.TextSortReducer.setup(TextSortReducer.java:105)
>
>         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)
>
>         at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
>
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
>
>
> Please let me know what could be the issue.
>
>
> Thanks,
> Jilani
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message