flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3230) Kinesis streaming producer
Date Tue, 19 Apr 2016 10:30:25 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247514#comment-15247514

ASF GitHub Bot commented on FLINK-3230:

Github user uce commented on the pull request:

    Great addition to have a producer to Kinesis. The changes look good overall, I had some
minor inline comments.
    The main concern is that we can only test this with a real Kinesis setup, which makes
it fragile for future changes etc. Did you consider trying to mock the `KinesisProducer` to
test that the expected calls are issued to it?
    PS: The title of the JIRA issue and this PR is confusing (the last `... into Flink` part)

> Kinesis streaming producer
> --------------------------
>                 Key: FLINK-3230
>                 URL: https://issues.apache.org/jira/browse/FLINK-3230
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Robert Metzger
> Add a FlinkKinesisProducer for the Flink Kinesis streaming connector. We will be using
AWS SDK implementation for code consistency with the FlinkKinesisConsumer.
> The features of FlinkKinesisProducer is rather straightforward:
> 1. Partition put records based on partition key.
> 2. Configurable put mode: Bulk put for higher throughput vs. sequential single record
puts. Size of bulk should also be configurable.
> 3. For bulk put, user can also choose to enforce strict ordering of the result with the
tradeoff of higher put latency. Ref: https://brandur.org/kinesis-order

This message was sent by Atlassian JIRA

View raw message