kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Spark SQL on kudu can not contains nullable columns?
Date Tue, 01 Mar 2016 17:29:06 GMT
Ha yeah that's a good one. I opened this jira:
https://issues.apache.org/jira/browse/KUDU-1360

Basically we forgot to check for nulls :)

J-D

On Tue, Mar 1, 2016 at 9:18 AM, Darren Hoo <darren.hoo@gmail.com> wrote:

> Spark SQL on kudu can not contains nullable columns?
>
> I've create one table in kudu(0.6.0) which has nullable columns,
> when I try to use spark sql (using kudu java client 0.7.0) like this:
>
> sqlContext.load("org.kududb.spark",Map("kudu.table" -> "contents",
> "kudu.master" -> "master1:7051")).registerTempTable("contents")
> sqlContext.sql("SELECT * FROM * FROM contents limit 10").collectAsList()
>
> I got this error:
>
> 16/03/02 00:45:42 INFO DAGScheduler: Job 4 failed: collect at
> <console>:20, took 11.813423 s
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage
> 7.0 (TID 62, slave29): java.lang.IllegalArgumentException: The requested
> column (4)  is null
>         at org.kududb.client.RowResult.checkNull(RowResult.java:475)
>         at org.kududb.client.RowResult.getString(RowResult.java:321)
>         at org.kududb.client.RowResult.getString(RowResult.java:308)
>         at org.kududb.spark.KuduRelation.org
> $kududb$spark$KuduRelation$$getKuduValue(DefaultSource.scala:144)
>         at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
>         at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
>         at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>         at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>         at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>         at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>         at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>         at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>         at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:126)
>         at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:124)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>         at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>         at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
>         at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
>         at scala.collection.TraversableOnce$class.to
> (TraversableOnce.scala:273)
>         at scala.collection.AbstractIterator.to(Iterator.scala:1157)
>         at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
>         at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
>         at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
>         at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
>         at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
>         at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
>         at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
>         at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
>         at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>         at org.apache.spark.scheduler.Task.run(Task.scala:88)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> Is this due to the version incompatibily between my kudu server(0.6.0) and
> java client (0.7.0)?
>
>

Mime
View raw message