spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran Rashid <iras...@cloudera.com>
Subject Re: Workaround for spark 1.2.X roaringbitmap kryo problem?
Date Thu, 12 Mar 2015 00:38:22 GMT
I'm not sure what you mean.   Are you asking how you can recompile all of
spark and deploy it, instead of using one of the pre-built versions?

https://spark.apache.org/docs/latest/building-spark.html

Or are you seeing compile problems specifically w/
HighlyCompressedMapStatus?   The code compiles fine, so I'm not sure what
problem you are running into -- we'd need a lot more info to help

On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.luthra@gmail.com> wrote:

> Does anyone know how to get the HighlyCompressedMapStatus to compile?
>
> I will try turning off kryo in 1.2.0 and hope things don't break.  I want
> to benefit from the MapOutputTracker fix in 1.2.0.
>
> On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <irashid@cloudera.com> wrote:
>
>> the scala syntax for arrays is Array[T], not T[], so you want to use
>> something:
>>
>> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
>> kryo.register(classOf[Array[Short]])
>>
>> nonetheless, the spark should take care of this itself.  I'll look into
>> it later today.
>>
>>
>> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.luthra@gmail.com>
>> wrote:
>>
>>> I think this is a Java vs scala syntax issue. Will check.
>>>
>>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com>
>>> wrote:
>>>
>>>> Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
>>>>
>>>> I tried this as a workaround:
>>>>
>>>> import org.apache.spark.scheduler._
>>>> import org.roaringbitmap._
>>>>
>>>> ...
>>>>
>>>>
>>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray])
>>>>     kryo.register(classOf[org.roaringbitmap.ArrayContainer])
>>>>
>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>>     kryo.register(classOf[short[]])
>>>>
>>>>
>>>> in build file:
>>>>
>>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"
>>>>
>>>>
>>>> This fails to compile:
>>>>
>>>> ...:53: identifier expected but ']' found.
>>>>
>>>> [error]
>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>>
>>>> also:
>>>>
>>>> :54: identifier expected but ']' found.
>>>>
>>>> [error]     kryo.register(classOf[short[]])
>>>> also:
>>>>
>>>> :51: class HighlyCompressedMapStatus in package scheduler cannot be
>>>> accessed in package org.apache.spark.scheduler
>>>> [error]
>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>>
>>>>
>>>> Suggestions?
>>>>
>>>> Arun
>>>>
>>>
>>>
>>
>

Mime
View raw message