hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Patil <vinay18.pa...@gmail.com>
Subject Re: [Discuss] Provide a Flag to choose between Flink or Spark
Date Sun, 27 Jun 2021 09:23:32 GMT
Thanks Danny and Vinoth for your comments.

I have created  https://issues.apache.org/jira/browse/HUDI-2082 for
tracking this.


Regards,
Vinay Patil


On Wed, Jun 16, 2021 at 12:53 PM Vinoth Chandar <vinoth@apache.org> wrote:

> +1 on this effort overall. It will be a little tricky, but doable.
>
> First thing is to see how we can replace raw usages of Spark APIs with
> HoodieEngineContext. It will be cool if we can completely generify
> DeltaStreamer.
> but I suspect we need flink/spark specific modules ultimately
>
> On Wed, Jun 16, 2021 at 12:17 AM Danny Chan <danny0405@apache.org> wrote:
>
> > There was actually an issue here:
> > https://issues.apache.org/jira/browse/HUDI-1872, maybe you can take it
> and
> > go on with the work ~
> >
> > Best,
> > Danny Chan
> >
> > Vinay Patil <vinay18.patil@gmail.com> 于2021年6月11日周五 下午3:26写道:
> >
> > > Thank you Danny for your response.
> > >
> > > Can we have a JIRA story where all the refactoring is required for
> > > Hudi-Flink code as well.
> > >
> > > I will create a task if we agree that a Flag will be helpful to choose
> > > different runners
> > >
> > > Regards,
> > > Vinay Patil
> > >
> > >
> > > On Fri, Jun 11, 2021 at 12:34 PM Danny Chan <danny0405@apache.org>
> > wrote:
> > >
> > > > Basically agree with that, but before that we may need some
> refactoring
> > > to
> > > > the existing code:
> > > >
> > > > Move the HoodieFlinkStreamer from the hudi-flink module into the
> > > > hudi-utilities to be together with the HoodieDeltaStreamer.
> > > > We are planning to add separate flink compaction programs too, which
> > has
> > > > the same problem.
> > > >
> > > > Best,
> > > > Danny Chan
> > > >
> > > > Vinay Patil <vinay18.patil@gmail.com> 于2021年6月9日周三 下午3:42写道:
> > > >
> > > > > Hi Team,
> > > > >
> > > > > Currently, Hudi supports Flink as well Spark, there are two
> different
> > > > > classes
> > > > > 1. HoodieDeltaStreamer
> > > > > 2. FlinkHoodieDeltaStreamer
> > > > >
> > > > > Should we have a provision to pass the flag like --runner to choose
> > > > between
> > > > > Flink or Spark and have a single entry point class which will take
> > all
> > > > the
> > > > > common configs.
> > > > >
> > > > > Based on the runner flag, we can call HoodieDeltaStreamer or
> > > > > FlinkHoodieDeltaStreamer
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Regards,
> > > > > Vinay Patil
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message