hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <chetan.opensou...@gmail.com>
Subject Re: Spark Streaming checkpoint recovery failure for HBaseContext
Date Fri, 27 Jan 2017 08:32:06 GMT
The hbase-spark module is not production ready , having some issues there
are TODO: patches - https://issues.apache.org/jira/browse/HBASE-16179
- https://issues.apache.org/jira/browse/HBASE-14160

But by changing according configuration and using build.sbt i got success
to get run it. would you share exact code block how you configured
hbasecontext and pom.xml


On Fri, Jan 27, 2017 at 11:53 AM, SivaRam Bollineni <
shivaram.bollineni@gmail.com> wrote:

> hi,
>
> First of all, Thanks a ton for HBaseContext API, it's very useful and
> performant for our use-cases. We use HBaseContext in Spark Streaming with
> checkpoint. We have a problem in recovering from checkpoint. Below is the
> *NullPointerException* stack trace. I am sure it's because SparkContext
> object is passed as null to HBaseContext. SparkContext is not serializable
> and without SparkContext we can't create HBaseContext. Please help me in
> resolving this issue.
>
> java.lang.NullPointerException
>
>         at
> com.cloudera.spark.hbase.HBaseContext.<init>(HBaseContext.scala:69)
>
>         at
> com.abc.bbc.xyz.MySparkStreaming$$anonfun$createStreamingContext$1.
> apply(MySparkStreaming.scala:118)
>
>         at com.abc.bbc.xyz. MySparkStreaming
> $$anonfun$createStreamingContext$1.apply(MySparkStreaming.scala:94)
>
>         at
> org.apache.spark.streaming.dstream.DStream$$anonfun$
> foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
>
>         at
> org.apache.spark.streaming.dstream.DStream$$anonfun$
> foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
>
>         at
> org.apache.spark.streaming.dstream.ForEachDStream$$
> anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:42)
>
>         at
> org.apache.spark.streaming.dstream.ForEachDStream$$
> anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
>
>         at
> org.apache.spark.streaming.dstream.ForEachDStream$$
> anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
>
>         at
> org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(
> DStream.scala:399)
>
>         at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(
> ForEachDStream.scala:40)
>
>         at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(
> ForEachDStream.scala:40)
>
>         at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(
> ForEachDStream.scala:40)
>
>         at scala.util.Try$.apply(Try.scala:161)
>
>         at org.apache.spark.streaming.scheduler.Job.run(Job.scala:34)
>
>         at
> org.apache.spark.streaming.scheduler.JobScheduler$
> JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:207)
>
>         at
> org.apache.spark.streaming.scheduler.JobScheduler$
> JobHandler$$anonfun$run$1.apply(JobScheduler.scala:207)
>
>         at
> org.apache.spark.streaming.scheduler.JobScheduler$
> JobHandler$$anonfun$run$1.apply(JobScheduler.scala:207)
>
>         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
>         at
> org.apache.spark.streaming.scheduler.JobScheduler$
> JobHandler.run(JobScheduler.scala:206)
>
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
>
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Thanks !
> Siva
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message