hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinoth Chandar (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HUDI-516) Avoid need to import spark-avro package when submitting Hudi job in spark
Date Sat, 07 Mar 2020 06:16:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053902#comment-17053902
] 

Vinoth Chandar commented on HUDI-516:
-------------------------------------

[~lamber-ken] this should be resolved now with your change.. correct? feel free to close this
out if so 

> Avoid need to import spark-avro package when submitting Hudi job in spark
> -------------------------------------------------------------------------
>
>                 Key: HUDI-516
>                 URL: https://issues.apache.org/jira/browse/HUDI-516
>             Project: Apache Hudi (incubating)
>          Issue Type: Improvement
>          Components: Usability
>            Reporter: Udit Mehrotra
>            Assignee: lamber-ken
>            Priority: Major
>             Fix For: 0.6.0
>
>
> We are in the process of migrating Hudi to *spark 2.4.4* and using *spark-avro* instead
of the deprecated *databricks-avro* here [https://github.com/apache/incubator-hudi/pull/1005/]
> After this change, users would be required to specifically download spark-avro while
start spark-shell using:
> {code:java}
> --packages org.apache.spark:spark-avro_2.11:2.4.4
> {code}
> This is because we are not shading this now in *hudi-spark-bundle*. One reason for not
shading this is because we are not sure of the implications of shading a spark dependency
in a jar which is being submitted to spark. [~vinoth] pointed out that a possible concern
could be that we will always be shading spark-avro 2.4.4 which can affect users using higher
versions of Spark.
> This Jira is to come up with a way to solve this usability issue.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message