hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinoth Chandar <vin...@apache.org>
Subject Re: [DISCUSS] Create Spark and Flink utilities module
Date Tue, 20 Jul 2021 16:16:18 GMT
Hi Vinay.

Thanks for kicking this off.

I wonder if it's possible to structure the code in separate modules, but
have a single bundle.
Or is that a painful experience? (if so, can you share what issues we are
running into?)

We have rarely done backwards incompatible changes and users appreciate
that.
So love to understand why this is warranted here.

Thanks
Vinoth

On Sat, Jul 17, 2021 at 7:58 AM Vinay Patil <vinay18.patil@gmail.com> wrote:

> Hi Team,
>
> As part of https://issues.apache.org/jira/browse/HUDI-1872, we are
> creating
> a separate flink-utilities module. Based on our discussion on the PR,
> should we even create a spark-utilities module. This would look like :
>
> hudi-utilities
> ├── hudi-flink-utilities
> └── hudi-spark-utilities
>
> This would also mean to create separate utilities-bundle for Flink and
> Spark,
>
> hudi-utilities-bundle
>  ├── hudi-flink-utilities-bundle
>  └── hudi-spark-utilities-bundle
>
> This is not a backward compatible change as users will have to provide an
> engine specific bundle. IMO, since Hudi is supporting Flink and Spark it
> will be good to have  engine specific bundle.
>
> What do you think?
>
> Regards,
> Vinay Patil
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message