hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinoth Chandar (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HUDI-519) Document the need for Avro dependency shading/relocation for custom payloads, need for spark-avro
Date Mon, 03 Feb 2020 18:16:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029154#comment-17029154
] 

Vinoth Chandar commented on HUDI-519:
-------------------------------------

[~xleesf] took care of this already .. 

> Document the need for Avro dependency shading/relocation for custom payloads, need for
spark-avro
> -------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-519
>                 URL: https://issues.apache.org/jira/browse/HUDI-519
>             Project: Apache Hudi (incubating)
>          Issue Type: Task
>          Components: Docs, Usability
>            Reporter: Udit Mehrotra
>            Assignee: leesf
>            Priority: Major
>             Fix For: 0.5.1
>
>
> In [https://github.com/apache/incubator-hudi/pull/1005] we are migrating Hudi to Spark
2.4.4. As part of this migration, we also had to migrate Hudi to use Avro 1.8.2 (required
by spark), while Hive still uses older version of Avro.
> This has resulted in the need to shade Avro in *hadoop-mr-bundle*. This has implications
on users of Hudi, who implement custom record payloads. They would have start shading Avro
in there custom jars, similar to how it shaded in *hadoop-mr-bundle*.
> This Jira is to track the documentation of this caveat in release notes, and if needed
at other places like website etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message