beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grzegorz Kołakowski (JIRA) <j...@apache.org>
Subject [jira] [Updated] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
Date Tue, 27 Feb 2018 14:24:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Grzegorz Kołakowski updated BEAM-3089:
--------------------------------------
    Attachment: flink-ui-parallelism.png

> Issue with setting the parallelism at client level using Flink runner
> ---------------------------------------------------------------------
>
>                 Key: BEAM-3089
>                 URL: https://issues.apache.org/jira/browse/BEAM-3089
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.0.0
>         Environment: I am using Flink 1.2.1 running on Docker, with Task Managers distributed
across different VMs as part of a Docker Swarm.
>            Reporter: Thalita Vergilio
>            Assignee: Aljoscha Krettek
>            Priority: Major
>              Labels: docker, flink, parallel-deployment
>         Attachments: flink-ui-parallelism.png
>
>
> When uploading an Apache Beam application using the Flink Web UI, the parallelism set
at job submission doesn't get picked up. The same happens when submitting a job using the
Flink CLI.
> In both cases, the parallelism ends up defaulting to 1.
> When I set the parallelism programmatically within the Apache Beam code, it works: {{flinkPipelineOptions.setParallelism(4);}}
> I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory
class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values
passed to Flink, then defaults to 1 if it doesn't find anything.
> Any ideas on how this could be fixed or worked around? I need to be able to change the
parallelism dynamically, so the programmatic approach won't really work for me, nor will setting
the Flink configuration at system level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message