beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía <ieme...@gmail.com>
Subject Re: Problem with Pipeline In Flink Runner
Date Fri, 20 May 2016 09:07:35 GMT
Hello again,

I just executed our pipeline with the changes you just merged, and guess
what ? It works perfectly (in batch mode), amazing job guys. I have just one
extra question, is there a way to configure the pipeline options with system
properties (-D...).  e.g. to configure the streaming/batch mode and avoid
to do
this explicitly in the code (to remove the explicit FlinkPipelineOptions
import) ?

Ismaël


On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <mxm@apache.org> wrote:

> Hi Ismael,
>
> Side inputs are not supported when the Flink Runner runs in streaming
> mode. Actually, we just merged changes which allow you to use the
> batch mode (streaming set to false) with Windows and side inputs.
>
> So just checkout the latest Beam master and rebuild. Set the streaming
> flag to false again and rerun your example.
>
> Thanks,
> Max
>
> On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <iemejia@gmail.com> wrote:
> >
> > Hello,
> >
> > Thanks for your answer, I am really happy to know you are working on
> > removing
> > this flag for batch/streaming since this is not really Beam-like.
> >
> > I tried to execute the pipeline again in streaming mode, and I got this
> > exception now:
> >
> > java.lang.UnsupportedOperationException: The transform
> > View.CreatePCollectionView is currently not supported.
> >
> > Any info about this one ?
> >
> > Thanks again,
> > Ismael
> >
> >
> > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <aljoscha@apache.org>
> > wrote:
> >>
> >> Hi,
> >> as an addition, there are plans to get rid of the "streaming" flag for
> all
> >> runners and determine the mode from the job.
> >>
> >> -Aljoscha
> >>
> >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org> wrote:
> >>>
> >>> Hi Ismaël,
> >>>
> >>> There is a pull request which enables Windows on batch. It will be
> >>> merged soon. In the meantime, please use the streaming execution mode
> >>> of the Flink Runner.
> >>>
> >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
> >>>
> >>> Cheers,
> >>> Max
> >>>
> >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <iemejia@gmail.com>
> wrote:
> >>> > I was trying to run a local Pipeline we have (basically a batch one
> we
> >>> > rewrote from TPC-H)  in all the possible runners, for the moment I
> had
> >>> > success with the DirectPipelineRunner, InProcessPipelineRunner and
> >>> > SparkPipelineRunner. However when I executed it in the
> >>> > FlinkPipelineRunner I
> >>> > got this exception:
> >>> >
> >>> > java.lang.UnsupportedOperationException: The transform Window.Into()
> is
> >>> > currently not supported.
> >>> >
> >>> > I thought it was probably a configuration error since I remember that
> >>> > in the
> >>> > flink runner we had to explicitly define if it was a batch or
> streaming
> >>> > job,
> >>> > but when I checked the options:
> >>> >
> >>> > Current Settings:
> >>> >   appName: TestQ08Task
> >>> >   filesToStage: ...
> >>> >   flinkMaster: [auto]
> >>> >   parallelism: 1
> >>> >   runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> >>> >   stableUniqueNames: WARNING
> >>> >   streaming: false
> >>> >
> >>> > I noticed I was in batch mode, is there something missing in the
> >>> > options
> >>> > configuration, or is it that the UnsoppertedOperation is a WIP or not
> >>> > supported at all by Flink ?
> >>> >
> >>> > And extra question, this option to define if it is in batch or stream
> >>> > mode
> >>> > is going to stay for long, can't be this inferred some how ?
> >>> >
> >>> > Thanks,
> >>> > -Ismaël
> >>> >
> >
> >
>

Mime
View raw message