beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <>
Subject [jira] [Commented] (BEAM-139) Print mode (batch/streaming) during translation
Date Tue, 22 Mar 2016 15:27:25 GMT


Davor Bonaci commented on BEAM-139:

We are hoping to be able to remove the "--streaming" flag altogether in the near future. With
the Runner API refactoring, a runner should be able to observe the pipeline graph in its entirety,
and automatically choose parameters like the execution mode, without them being specified
the user.

For example, if there are unbounded PCollections in the graph, a runner may choose the streaming
execution mode. Note, however, that a runner may choose "micro-batch" too. Essentially, this
is a runner-specific decision that with a better Runner API, each runner can do unilaterally,
without the concept being exposed to the user.

The current "--streaming" flag is an unfortunate consequence of the existing runner API. In
that sense, I'd close this issue as we think this will go away shortly, and doesn't need to
be printed.

Sounds good?

> Print mode (batch/streaming) during translation 
> ------------------------------------------------
>                 Key: BEAM-139
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-core, runner-flink
>            Reporter: Maximilian Michels
>            Assignee: Davor Bonaci
> Runners have different feature sets for batch and streaming. It may be a good idea to
print/log the translation mode during parsing of the options in the {{PipelineOptionsFactory}}.
That would help users to understand they are missing the streaming flag in the options and
that the default mode is batch.

This message was sent by Atlassian JIRA

View raw message