hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Chan <danny0...@apache.org>
Subject Re: [Discuss] Provide a Flag to choose between Flink or Spark
Date Wed, 16 Jun 2021 07:17:11 GMT
There was actually an issue here:
https://issues.apache.org/jira/browse/HUDI-1872, maybe you can take it and
go on with the work ~

Best,
Danny Chan

Vinay Patil <vinay18.patil@gmail.com> 于2021年6月11日周五 下午3:26写道:

> Thank you Danny for your response.
>
> Can we have a JIRA story where all the refactoring is required for
> Hudi-Flink code as well.
>
> I will create a task if we agree that a Flag will be helpful to choose
> different runners
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jun 11, 2021 at 12:34 PM Danny Chan <danny0405@apache.org> wrote:
>
> > Basically agree with that, but before that we may need some refactoring
> to
> > the existing code:
> >
> > Move the HoodieFlinkStreamer from the hudi-flink module into the
> > hudi-utilities to be together with the HoodieDeltaStreamer.
> > We are planning to add separate flink compaction programs too, which has
> > the same problem.
> >
> > Best,
> > Danny Chan
> >
> > Vinay Patil <vinay18.patil@gmail.com> 于2021年6月9日周三 下午3:42写道:
> >
> > > Hi Team,
> > >
> > > Currently, Hudi supports Flink as well Spark, there are two different
> > > classes
> > > 1. HoodieDeltaStreamer
> > > 2. FlinkHoodieDeltaStreamer
> > >
> > > Should we have a provision to pass the flag like --runner to choose
> > between
> > > Flink or Spark and have a single entry point class which will take all
> > the
> > > common configs.
> > >
> > > Based on the runner flag, we can call HoodieDeltaStreamer or
> > > FlinkHoodieDeltaStreamer
> > >
> > > Thoughts?
> > >
> > > Regards,
> > > Vinay Patil
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message