hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Patil <vinay18.pa...@gmail.com>
Subject Re: [DISCUSS] Create Spark and Flink utilities module
Date Tue, 20 Jul 2021 16:34:59 GMT
Hi Vinoth,

> I wonder if it's possible to structure the code in separate modules, but
have a single bundle

Yes, this is possible, initially I started doing the same in this PR -
https://github.com/apache/hudi/pull/3162 , hence wanted to discuss this
here, if we create a single bundle, we have to make sure there are no
dependency conflicts. For example: Flink-Hudi is using a different version
of parquet as compared to Spark-Hudi because of which the tests started to
fail with `java.lang.NoSuchMethodError:
org.apache.parquet.column.ParquetProperties.getColumnIndexTruncateLength()`


Regards,
Vinay Patil


On Tue, Jul 20, 2021 at 9:46 PM Vinoth Chandar <vinoth@apache.org> wrote:

> Hi Vinay.
>
> Thanks for kicking this off.
>
> I wonder if it's possible to structure the code in separate modules, but
> have a single bundle.
> Or is that a painful experience? (if so, can you share what issues we are
> running into?)
>
> We have rarely done backwards incompatible changes and users appreciate
> that.
> So love to understand why this is warranted here.
>
> Thanks
> Vinoth
>
> On Sat, Jul 17, 2021 at 7:58 AM Vinay Patil <vinay18.patil@gmail.com>
> wrote:
>
> > Hi Team,
> >
> > As part of https://issues.apache.org/jira/browse/HUDI-1872, we are
> > creating
> > a separate flink-utilities module. Based on our discussion on the PR,
> > should we even create a spark-utilities module. This would look like :
> >
> > hudi-utilities
> > ├── hudi-flink-utilities
> > └── hudi-spark-utilities
> >
> > This would also mean to create separate utilities-bundle for Flink and
> > Spark,
> >
> > hudi-utilities-bundle
> >  ├── hudi-flink-utilities-bundle
> >  └── hudi-spark-utilities-bundle
> >
> > This is not a backward compatible change as users will have to provide an
> > engine specific bundle. IMO, since Hudi is supporting Flink and Spark it
> > will be good to have  engine specific bundle.
> >
> > What do you think?
> >
> > Regards,
> > Vinay Patil
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message