beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "peay (JIRA)" <>
Subject [jira] [Created] (BEAM-2449) Support ValueProviders in KafkaIO
Date Wed, 14 Jun 2017 16:44:00 GMT
peay created BEAM-2449:

             Summary: Support ValueProviders in KafkaIO
                 Key: BEAM-2449
             Project: Beam
          Issue Type: Improvement
          Components: runner-dataflow, sdk-java-extensions
    Affects Versions: 2.0.0
            Reporter: peay
            Assignee: Thomas Groh
            Priority: Minor

KafkaIO takes parameters at construction time:
- When partitions are provided, KafkaIO actually does nothing particular at construction time.
- When only a topic is provided, I believe KafkaIO does fetch the list of partitions at construction

Using ValueProvider for the brokers would be a good improvement. In particular, when partitions
are provided, a RuntimeProvider could be passed in.

Being able to always use a RuntimeProvider would be even better, such as when using {{}}:
this would considerably improve the workflow of using Dataflow templates with KafkaIO, which
currently requires to hardcode brokers into the template.

This message was sent by Atlassian JIRA

View raw message