spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From roywang1024 <roywang1...@163.com>
Subject Re:Re: Re: Re: Error in load hbase on spark
Date Sat, 10 Oct 2015 01:26:02 GMT


Finally I fix it .
It just cause by "ClassNotFoundException: org.apache.htrace.Trace".
I can't see this message in logs on driver node,but can be found on worker node.
And I modify "spark.executor.extraClassPath" in spark-default.conf still not work.Also modify
classpath.txt on every node.
It can work use spark-submit options "--driver-java-options -Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar"


Thanks.



At 2015-10-09 21:10:22, "Ted Yu" <yuzhihong@gmail.com> wrote:

Can you pastebin log snippet showing hbase related errors ?


Please also consider posting the question on vendor's forum.



On Thu, Oct 8, 2015 at 10:17 PM, roywang1024 <roywang1024@163.com> wrote:


I add hbase-conf-dir to spark/conf/classpath.txt,but still error.







At 2015-10-09 11:04:35, "Ted Yu" <yuzhihong@gmail.com> wrote:

The second code snippet is similar to:

examples//src/main/scala/org/apache/spark/examples/HBaseTest.scala



See the comment in HBaseTest.scala :
    // please ensure HBASE_CONF_DIR is on classpath of spark driver
    // e.g: set it through spark.driver.extraClassPath property
    // in spark-defaults.conf or through --driver-class-path
    // command line option of spark-submit

If during execution of TableInputFormatBase#initializeTable(), there was exception, table
field might not have been initialized.

FYI



On Thu, Oct 8, 2015 at 7:54 PM, roywang1024 <roywang1024@163.com> wrote:


I have try this


SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
Configuration conf = HBaseConfiguration.create();
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes("InnerCode"));
scan.addColumn(Bytes.toBytes("InnerCode"), Bytes.toBytes(""));
conf.set(TableInputFormat.INPUT_TABLE, "SecuMain");
conf.set(TableInputFormat.SCAN, convertScanToString(scan));
and this


SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
Configuration conf = HBaseConfiguration.create();
String tableName = "SecuMain";
conf.set(TableInputFormat.INPUT_TABLE, tableName);
also can't wok!

Should I add hbase-site.xml to conf?

Thanks.








At 2015-10-09 10:35:16, "Ted Yu" <yuzhihong@gmail.com> wrote:

One possibility was that hbase config, including hbase.zookeeper.quorum, was not passed to
your job.
hbase-site.xml should be on the classpath.


Can you show snippet of your code ?


Looks like you were running against hbase 1.x


Cheers


On Thu, Oct 8, 2015 at 7:29 PM, Roy Wang <roywang1024@163.com> wrote:

I want to load hbase table into spark.
JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD =
sc.newAPIHadoopRDD(conf, TableInputFormat.class,
ImmutableBytesWritable.class, Result.class);

*when call hBaseRDD.count(),got error.*

Caused by: java.lang.IllegalStateException: The input format instance has
not been properly initialized. Ensure you call initializeTable either in
your constructor or initialize method
        at
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:389)
        at
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:158)
        ... 11 more

*But when job start,I can get these logs*
2015-10-09 09:17:00[main] WARN  TableInputFormatBase:447 - initializeTable
called multiple times. Overwriting connection and table reference;
TableInputFormatBase will not close these old references when done.

Does anyone know how does this happen?

Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Error-in-load-hbase-on-spark-tp24986.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org








 







 


Mime
View raw message