beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thalita Vergilio (JIRA)" <>
Subject [jira] [Created] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
Date Sun, 22 Oct 2017 17:53:00 GMT
Thalita Vergilio created BEAM-3089:

             Summary: Issue with setting the parallelism at client level using Flink runner
                 Key: BEAM-3089
             Project: Beam
          Issue Type: Bug
          Components: runner-flink
    Affects Versions: 2.0.0
         Environment: I am using Flink 1.2.1 running on Docker, with Task Managers distributed
across different VMs as part of a Docker Swarm.
            Reporter: Thalita Vergilio
            Assignee: Aljoscha Krettek

When uploading an Apache Beam application using the Flink Web UI, the parallelism set at job
submission doesn't get picked up. The same happens when submitting a job using the Flink CLI.

In both cases, the parallelism ends up defaulting to 1.

When I set the parallelism programmatically within the Apache Beam code, it works: {{flinkPipelineOptions.setParallelism(4);}}

I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory
class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values
passed to Flink, then defaults to 1 if it doesn't find anything.

Any ideas on how this could be fixed or worked around? I need to be able to change the parallelism
dynamically, so the programmatic approach won't really work for me, nor will setting the Flink
configuration at system level.

This message was sent by Atlassian JIRA

View raw message