hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Patil <vinay18.pa...@gmail.com>
Subject Re: [Discuss] Provide a Flag to choose between Flink or Spark
Date Fri, 11 Jun 2021 07:25:46 GMT
Thank you Danny for your response.

Can we have a JIRA story where all the refactoring is required for
Hudi-Flink code as well.

I will create a task if we agree that a Flag will be helpful to choose
different runners

Regards,
Vinay Patil


On Fri, Jun 11, 2021 at 12:34 PM Danny Chan <danny0405@apache.org> wrote:

> Basically agree with that, but before that we may need some refactoring to
> the existing code:
>
> Move the HoodieFlinkStreamer from the hudi-flink module into the
> hudi-utilities to be together with the HoodieDeltaStreamer.
> We are planning to add separate flink compaction programs too, which has
> the same problem.
>
> Best,
> Danny Chan
>
> Vinay Patil <vinay18.patil@gmail.com> 于2021年6月9日周三 下午3:42写道:
>
> > Hi Team,
> >
> > Currently, Hudi supports Flink as well Spark, there are two different
> > classes
> > 1. HoodieDeltaStreamer
> > 2. FlinkHoodieDeltaStreamer
> >
> > Should we have a provision to pass the flag like --runner to choose
> between
> > Flink or Spark and have a single entry point class which will take all
> the
> > common configs.
> >
> > Based on the runner flag, we can call HoodieDeltaStreamer or
> > FlinkHoodieDeltaStreamer
> >
> > Thoughts?
> >
> > Regards,
> > Vinay Patil
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message