beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <>
Subject [jira] [Commented] (BEAM-137) Add implicit conf/pipeline-default.conf options file
Date Mon, 21 Mar 2016 17:17:25 GMT


Luke Cwik commented on BEAM-137:

The original design of PipelineOptions took into account the need for a multi-runner / options
world (in Java) since it took onto itself a runner/options registration mechanism. We support
registration via Java's ServiceLoader so just having the relevant jar files on the classpath
will have those additional PipelineOptions / Runners available. We already use this registration
mechanism for Dataflow and other runners should utilize it as well (if they haven't already).

I think this feature request would be as simple as something like .fromFile(String path) as
an addition to .fromArgs(String[] args) and the order in which they are invoked will state
what overrides what. The .fromFile could use the json format we already rely on for serializing

Or, there can be a special flag similar to "runner" which gets handled inside PipelineOptionsFactory.
It would load the options from the config file and use options specified on the command line
as overrides.

> Add implicit conf/pipeline-default.conf options file
> ----------------------------------------------------
>                 Key: BEAM-137
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Jean-Baptiste Onofré
>            Assignee: Davor Bonaci
> Right now, most of users provide the pipeline options via the main arguments.
> For instance, it's the classic way to provide pipeline input, etc.
> For convenience, it would be great that the pipeline looks for options in conf/[pipeline_name]-default.conf
by default, and override the options using the main arguments.
> Thoughts ?

This message was sent by Atlassian JIRA

View raw message