beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: Problem with Pipeline In Flink Runner
Date Fri, 20 May 2016 11:59:43 GMT
You can always do it via System, populating the options, right ?

Regards
JB

On 05/20/2016 11:07 AM, Ismaël Mejía wrote:
> Hello again,
>
> I just executed our pipeline with the changes you just merged, and guess
> what ? It works perfectly (in batch mode), amazing job guys. I have just one
> extra question, is there a way to configure the pipeline options with system
> properties (-D...).  e.g. to configure the streaming/batch mode and
> avoid to do
> this explicitly in the code (to remove the explicit FlinkPipelineOptions
> import) ?
>
> Ismaël
>
>
> On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <mxm@apache.org
> <mailto:mxm@apache.org>> wrote:
>
>     Hi Ismael,
>
>     Side inputs are not supported when the Flink Runner runs in streaming
>     mode. Actually, we just merged changes which allow you to use the
>     batch mode (streaming set to false) with Windows and side inputs.
>
>     So just checkout the latest Beam master and rebuild. Set the streaming
>     flag to false again and rerun your example.
>
>     Thanks,
>     Max
>
>     On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <iemejia@gmail.com
>     <mailto:iemejia@gmail.com>> wrote:
>      >
>      > Hello,
>      >
>      > Thanks for your answer, I am really happy to know you are working on
>      > removing
>      > this flag for batch/streaming since this is not really Beam-like.
>      >
>      > I tried to execute the pipeline again in streaming mode, and I
>     got this
>      > exception now:
>      >
>      > java.lang.UnsupportedOperationException: The transform
>      > View.CreatePCollectionView is currently not supported.
>      >
>      > Any info about this one ?
>      >
>      > Thanks again,
>      > Ismael
>      >
>      >
>      > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek
>     <aljoscha@apache.org <mailto:aljoscha@apache.org>>
>      > wrote:
>      >>
>      >> Hi,
>      >> as an addition, there are plans to get rid of the "streaming"
>     flag for all
>      >> runners and determine the mode from the job.
>      >>
>      >> -Aljoscha
>      >>
>      >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org
>     <mailto:mxm@apache.org>> wrote:
>      >>>
>      >>> Hi Ismaël,
>      >>>
>      >>> There is a pull request which enables Windows on batch. It will be
>      >>> merged soon. In the meantime, please use the streaming
>     execution mode
>      >>> of the Flink Runner.
>      >>>
>      >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
>      >>>
>      >>> Cheers,
>      >>> Max
>      >>>
>      >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía
>     <iemejia@gmail.com <mailto:iemejia@gmail.com>> wrote:
>      >>> > I was trying to run a local Pipeline we have (basically a
>     batch one we
>      >>> > rewrote from TPC-H)  in all the possible runners, for the
>     moment I had
>      >>> > success with the DirectPipelineRunner,
>     InProcessPipelineRunner and
>      >>> > SparkPipelineRunner. However when I executed it in the
>      >>> > FlinkPipelineRunner I
>      >>> > got this exception:
>      >>> >
>      >>> > java.lang.UnsupportedOperationException: The transform
>     Window.Into() is
>      >>> > currently not supported.
>      >>> >
>      >>> > I thought it was probably a configuration error since I
>     remember that
>      >>> > in the
>      >>> > flink runner we had to explicitly define if it was a batch or
>     streaming
>      >>> > job,
>      >>> > but when I checked the options:
>      >>> >
>      >>> > Current Settings:
>      >>> >   appName: TestQ08Task
>      >>> >   filesToStage: ...
>      >>> >   flinkMaster: [auto]
>      >>> >   parallelism: 1
>      >>> >   runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
>      >>> >   stableUniqueNames: WARNING
>      >>> >   streaming: false
>      >>> >
>      >>> > I noticed I was in batch mode, is there something missing in the
>      >>> > options
>      >>> > configuration, or is it that the UnsoppertedOperation is a
>     WIP or not
>      >>> > supported at all by Flink ?
>      >>> >
>      >>> > And extra question, this option to define if it is in batch
>     or stream
>      >>> > mode
>      >>> > is going to stay for long, can't be this inferred some how ?
>      >>> >
>      >>> > Thanks,
>      >>> > -Ismaël
>      >>> >
>      >
>      >
>
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message