spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Han JU <ju.han.fe...@gmail.com>
Subject Re: System property for Kryo's setRegistrationRequired
Date Thu, 17 Oct 2013 21:57:09 GMT
Matei, thanks for the reply.
I've tried setting kryo.setRegistrationOptional by KryoRegistrator but it
seems that some spark's internal class, such as
o.a.spark.scheduler.MapStatus, need to be registered.


2013/10/17 Matei Zaharia <matei.zaharia@gmail.com>

> This is a good idea, though you may also be able to do it in your
> KryoRegistrator by just calling kryo.setRegistrationOptional there. (You
> get full access to the Kryo object.)
>
> Omitting to register a class is not an error by the way, it just leads to
> bigger data.
>
> Matei
>
> On Oct 16, 2013, at 3:28 AM, Han JU <ju.han.felix@gmail.com> wrote:
>
> > Hi,
> >
> > I think maybe it's better to have an option to force class registration
> > when using Kryo, mainly to prevent human errors.
> > Kryo has a setRegistrationRequired method and by default the check is
> false
> > (in kryo and in chill). Maybe we can have a system property for setting
> > this? Like:
> >
> > System.setProperty("spark.kryo.registrationRequired", true)
> >
> > What you say about this?
> > --
> > *JU Han*
> >
> > Data Engineer @ Botify.com
> >
> > +33 0619608888
>
>


-- 
*JU Han*

Data Engineer @ Botify.com

+33 0619608888

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message