spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From RotemShaul <>
Subject [GitHub] spark issue #13761: [SPARK-12197] [SparkCore] Kryo & Avro - Support Schema R...
Date Sat, 23 Jul 2016 02:38:50 GMT
Github user RotemShaul commented on the issue:
    Indeed it is, but then you lose the already implemented
    GenericAvroSerializer abilities which come out of the box with Spark.
    (Caching / Registering of static schemas )
    As Spark already chose to (partially) support Avro from within SparkCore,
    to me it makes sense it will also support schema repos, as they are very
    common with Avro users to deal with Schema Evolution.
    It was this partial support that actually sparked the idea of 'if they
    support registering of Avro schemas, why not go all the way  ?' and that's
    why I created the PR in the first place.
    Avro Generic Records and Spark-Core users will always face the
    serialization problem of schemas, some might be able to solve it with
    static schemas, and other will need the dynamic solution. It makes sense
    that either SparkCore will provide solution for both use cases or none of
    them. (and let it be resolved by custom serializer)
    Just my opinion. In my current workplace - I took your
    GenericAvroSerializer, added few lines of code to it, and used it as custom
    serializer. But it could be generalized - hence the PR.
    On Sat, Jul 23, 2016 at 5:14 AM, Reynold Xin <>
    > @RotemShaul <> is this something doable by
    > implementing a custom serializer outside Spark?
    > —
    > You are receiving this because you were mentioned.
    > Reply to this email directly, view it on GitHub
    > <>, or mute
    > the thread
    > <>
    > .

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message