beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Damien GOUYETTE (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2470) Inconsistent behavior on the functioning of the dataflow templates?
Date Mon, 19 Jun 2017 15:30:00 GMT
Damien GOUYETTE created BEAM-2470:
-------------------------------------

             Summary: Inconsistent behavior on the functioning of the dataflow templates?
                 Key: BEAM-2470
                 URL: https://issues.apache.org/jira/browse/BEAM-2470
             Project: Beam
          Issue Type: Bug
          Components: runner-core
    Affects Versions: 0.6.0
            Reporter: Damien GOUYETTE
            Assignee: Kenneth Knowles



0
down vote
favorite
When i create a dataflow template, the characteristics of Runtime parameters are not persisted
in the template file. At runtime, if i try to pass a value for this parameter, i take a 400
error

I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).

My parameters are the following :

trait MyParameters extends PipelineOptions {

  def getInput: ValueProvider[String]
  def setInput(value: ValueProvider[String]): Unit
}
They are registred with this code

val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
To create the template i call sbt with this parameters :

run-main jobs.XmlImportJob    --runner=DataflowRunner --project=MyProject  --templateLocation=gs://myBucket/XmlImportTemplate
 --tempLocation=gs://myBucket/staging --instance=myInstance
If i pass explicitly --input, it becomes a StaticValue instead of RuntimeValue, and this time,
i can see it in the template file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message