hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinoth Chandar <vin...@apache.org>
Subject Re: [Discuss] Provide a Flag to choose between Flink or Spark
Date Wed, 16 Jun 2021 07:22:54 GMT
+1 on this effort overall. It will be a little tricky, but doable.

First thing is to see how we can replace raw usages of Spark APIs with
HoodieEngineContext. It will be cool if we can completely generify
DeltaStreamer.
but I suspect we need flink/spark specific modules ultimately

On Wed, Jun 16, 2021 at 12:17 AM Danny Chan <danny0405@apache.org> wrote:

> There was actually an issue here:
> https://issues.apache.org/jira/browse/HUDI-1872, maybe you can take it and
> go on with the work ~
>
> Best,
> Danny Chan
>
> Vinay Patil <vinay18.patil@gmail.com> 于2021年6月11日周五 下午3:26写道:
>
> > Thank you Danny for your response.
> >
> > Can we have a JIRA story where all the refactoring is required for
> > Hudi-Flink code as well.
> >
> > I will create a task if we agree that a Flag will be helpful to choose
> > different runners
> >
> > Regards,
> > Vinay Patil
> >
> >
> > On Fri, Jun 11, 2021 at 12:34 PM Danny Chan <danny0405@apache.org>
> wrote:
> >
> > > Basically agree with that, but before that we may need some refactoring
> > to
> > > the existing code:
> > >
> > > Move the HoodieFlinkStreamer from the hudi-flink module into the
> > > hudi-utilities to be together with the HoodieDeltaStreamer.
> > > We are planning to add separate flink compaction programs too, which
> has
> > > the same problem.
> > >
> > > Best,
> > > Danny Chan
> > >
> > > Vinay Patil <vinay18.patil@gmail.com> 于2021年6月9日周三 下午3:42写道:
> > >
> > > > Hi Team,
> > > >
> > > > Currently, Hudi supports Flink as well Spark, there are two different
> > > > classes
> > > > 1. HoodieDeltaStreamer
> > > > 2. FlinkHoodieDeltaStreamer
> > > >
> > > > Should we have a provision to pass the flag like --runner to choose
> > > between
> > > > Flink or Spark and have a single entry point class which will take
> all
> > > the
> > > > common configs.
> > > >
> > > > Based on the runner flag, we can call HoodieDeltaStreamer or
> > > > FlinkHoodieDeltaStreamer
> > > >
> > > > Thoughts?
> > > >
> > > > Regards,
> > > > Vinay Patil
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message