beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía <ieme...@gmail.com>
Subject Re: Example: pass Runner at command line
Date Tue, 26 Jul 2016 07:28:10 GMT
Hello,

You can look at the repository for GDELT based examples that JB and me are
working on, they include maven profiles to run the Pipelines in most of the
existing runners (not Gearpump yet but soon). You can take those profiles
as an example. I just changed the README so you can see how to execute them
with Maven.

https://github.com/jbonofre/beam-samples

Ismaël





On Tue, Jul 26, 2016 at 12:30 AM, amir bahmanyari <amirtousa@yahoo.com>
wrote:

> Thanks Emanuele,
> Yes, I know these examples exist.
> I thought there has been one put together that addresses the Runner
> agnostic coding specifically without doing any extra work.
> A true "unified" example.
> Did you solve your State issue? I had the same questions sometime ago.
> For now, I use Redis to persist run-time state. Kinda poor man's way :-)
> works for now, but doesn't scale as I want it.
> Cheers
>
> ------------------------------
> *From:* Emanuele Cesena <emanuele@shopkick.com>
> *To:* user@beam.incubator.apache.org; amir bahmanyari <amirtousa@yahoo.com>
>
> *Sent:* Monday, July 25, 2016 3:18 PM
> *Subject:* Re: Example: pass Runner at command line
>
> Hi Amir,
>
> If you’re looking for a runner-independent example, you can find some in
> examples/.
>
> If you’re looking for runner-independent code that works on Flink, I think
> you should still wait a few iterations.
>
> Or, you can take code that refers to Flink, and remove the dependencies
> yourself.
>
> For instance, if you start from this example:
>
> https://github.com/ecesena/beam-starter/blob/master/src/main/java/com/dataradiant/beam/examples/WordCount.java
>
> You can remove the setRunner by passing -Drunner=FlinkRunner in pom.xml
> You can remove FlinkPipelineOptions but you have to implement yourself
> some of the getter/setter required by Flink, e.g. jobName:
>
> https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
>
> Hope this helps,
>
>
> > On Jul 25, 2016, at 2:59 PM, amir bahmanyari <amirtousa@yahoo.com>
> wrote:
> >
> > Hi Colleagues,
> > Is there a simple genetic example where the Runner is passed at the
> command line, the Beam code sets it in the generic Beam Options.set
> Runner(), and Pipeline.create() is?
> > No mention of  ANY specific Runner in the code like FlinkPipelineOptions
> .
> >
> > Thanks.
> > Amir-
>
>
> --
> Emanuele Cesena, Data Eng.
> http://www.shopkick.com
>
> Il corpo non ha ideali
>
>
>
>
>
>
>

Mime
View raw message