spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Luthra <arun.lut...@gmail.com>
Subject Re: Workaround for spark 1.2.X roaringbitmap kryo problem?
Date Mon, 02 Mar 2015 20:55:11 GMT
I think this is a Java vs scala syntax issue. Will check.

On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com> wrote:

> Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
>
> I tried this as a workaround:
>
> import org.apache.spark.scheduler._
> import org.roaringbitmap._
>
> ...
>
>
> kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
>     kryo.register(classOf[org.roaringbitmap.RoaringArray])
>     kryo.register(classOf[org.roaringbitmap.ArrayContainer])
>
> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>     kryo.register(classOf[short[]])
>
>
> in build file:
>
> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"
>
>
> This fails to compile:
>
> ...:53: identifier expected but ']' found.
>
> [error]
> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>
> also:
>
> :54: identifier expected but ']' found.
>
> [error]     kryo.register(classOf[short[]])
> also:
>
> :51: class HighlyCompressedMapStatus in package scheduler cannot be
> accessed in package org.apache.spark.scheduler
> [error]
> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>
>
> Suggestions?
>
> Arun
>

Mime
View raw message