beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Damien GOUYETTE (JIRA)" <>
Subject [jira] [Created] (BEAM-2470) Inconsistent behavior on the functioning of the dataflow templates?
Date Mon, 19 Jun 2017 15:30:00 GMT
Damien GOUYETTE created BEAM-2470:

             Summary: Inconsistent behavior on the functioning of the dataflow templates?
                 Key: BEAM-2470
             Project: Beam
          Issue Type: Bug
          Components: runner-core
    Affects Versions: 0.6.0
            Reporter: Damien GOUYETTE
            Assignee: Kenneth Knowles

down vote
When i create a dataflow template, the characteristics of Runtime parameters are not persisted
in the template file. At runtime, if i try to pass a value for this parameter, i take a 400

I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).

My parameters are the following :

trait MyParameters extends PipelineOptions {

  def getInput: ValueProvider[String]
  def setInput(value: ValueProvider[String]): Unit
They are registred with this code

val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
To create the template i call sbt with this parameters :

run-main jobs.XmlImportJob    --runner=DataflowRunner --project=MyProject  --templateLocation=gs://myBucket/XmlImportTemplate
 --tempLocation=gs://myBucket/staging --instance=myInstance
If i pass explicitly --input, it becomes a StaticValue instead of RuntimeValue, and this time,
i can see it in the template file.

This message was sent by Atlassian JIRA

View raw message