flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8267) update Kinesis Producer example for setting Region key
Date Mon, 15 Jan 2018 23:51:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326639#comment-16326639

ASF GitHub Bot commented on FLINK-8267:

GitHub user bowenli86 opened a pull request:


    [FLINK-8267] [Kinesis Connector] update Kinesis Producer example for setting Region key

    ## What is the purpose of the change
    Update doc to guide users to use KPL's native keys to set regions. 
    This originates from a bug that we forgot to set region keys explicitly for kinesis connector,
which has been fixed. According to the previous discussion between @tzulitai and I, we want
to remove AWSConfigConstants in 2.0 because it basically copies/translates config keys of
KPL/KCL, which doesn't add much value. 
    Guide users to use KPL's native keys to set regions can be the first step that facilitates
the migration.
    ## Brief change log
    - updated doc
    ## Verifying this change
    This change is a trivial rework / code cleanup without any test coverage.
    ## Does this pull request potentially affect one of the following parts:
    ## Documentation
      - Does this pull request introduce a new feature? (no)
    cc @tzulitai 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bowenli86/flink FLINK-8267

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5301
commit b4034f67e727fef68740221e3b31cd131c905df1
Author: Bowen Li <bowenli86@...>
Date:   2018-01-02T19:21:28Z

    update local branch

commit 11e9c255cd51304c5281d55226fbb6fe8360d8a2
Author: Bowen Li <bowenli86@...>
Date:   2018-01-04T01:35:11Z

    remove sh

commit e322a5416b0f4f89c366b98bb3571fbf6b7d460a
Author: Bowen Li <bowenli86@...>
Date:   2017-12-17T06:18:55Z

    update doc

commit 1b447633df4a8bfe7c4c19e7ae91aab6157756d7
Author: Bowen Li <bowenli86@...>
Date:   2018-01-15T23:48:38Z

    format code snippet

commit 4ede4b5a89d4bfcda8dcc845ab1da42177d22358
Author: Bowen Li <bowenli86@...>
Date:   2018-01-15T23:49:37Z

    remove ';' from scala code


> update Kinesis Producer example for setting Region key
> ------------------------------------------------------
>                 Key: FLINK-8267
>                 URL: https://issues.apache.org/jira/browse/FLINK-8267
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>            Reporter: Dyana Rose
>            Assignee: Bowen Li
>            Priority: Minor
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kinesis.html#kinesis-producer
> In the example code for the kinesis producer the region key is set like:
> {code:java}
> producerConfig.put(AWSConfigConstants.AWS_REGION, "us-east-1");
> {code}
> However, the AWS Kinesis Producer Library requires that the region key be Region (https://github.com/awslabs/amazon-kinesis-producer/blob/94789ff4bb2f5dfa05aafe2d8437d3889293f264/java/amazon-kinesis-producer-sample/default_config.properties#L269)
so the setting at this point should be:
> {code:java}
> producerConfig.put(AWSConfigConstants.AWS_REGION, "us-east-1");
> producerConfig.put("Region", "us-east-1");
> {code}
> When you run the Kinesis Producer you can see the effect of not setting the Region key
by a log line
> {noformat}
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer  - Started Kinesis
producer instance for region ''
> {noformat}
> The KPL also then assumes it's running on EC2 and attempts to determine it's own region,
which fails.
> {noformat}
> (EC2MetadataClient)Http request to Ec2MetadataService failed.
> [error] [main.cc:266] Could not configure the region. It was not given in the config
and we were unable to retrieve it from EC2 metadata
> {noformat}
> At the least I'd say the documentation should mention the difference between these two
keys and when you are required to also set the Region key.
> On the other hand, is this even the intended behaviour of this connector? Was it intended
that the AWSConfigConstants.AWS_REGION key also set the region of the of the kinesis stream?
The documentation for the example states 
> {noformat}
> The example demonstrates producing a single Kinesis stream in the AWS region “us-east-1”.
> {noformat}

This message was sent by Atlassian JIRA

View raw message