flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-6503) Refactor KinesisDataFetcher to separate concerns for shard discovery
Date Tue, 09 May 2017 08:19:04 GMT

     [ https://issues.apache.org/jira/browse/FLINK-6503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tzu-Li (Gordon) Tai updated FLINK-6503:
---------------------------------------
    Description: 
Currently, shard discovery is done within the {{KinesisDataFetcher}} 's loop. I propose to
extract that from the fetcher and encapsulate that in a separate class to separate concerns.

As can be seen in https://github.com/apache/flink/pull/3001/files#diff-0b702244caed9a73b7b3e1dc8a9cbaebR461,
one downside if we do not do this refactor is apparent when adding rescalability of the Kinesis
consumer. That required shards lookup operations prioir to running the fetcher, which resulted
in the need to expose shard fetching logic external to the `KinesisDataFetcher`. This should
be properly done be separating concerns of shard discovery from the fetcher.

  was:
Currently, shard discovery is done within the `KinesisDataFetcher`'s loop. I propose to extract
that from the fetcher and encapsulate that in a separate class to separate concerns.

As can be seen in https://github.com/apache/flink/pull/3001/files#diff-0b702244caed9a73b7b3e1dc8a9cbaebR461,
one downside if we do not do this refactor is apparent when adding rescalability of the Kinesis
consumer. That required shards lookup operations prioir to running the fetcher, which resulted
in the need to expose shard fetching logic external to the `KinesisDataFetcher`. This should
be properly done be separating concerns of shard discovery from the fetcher.


> Refactor KinesisDataFetcher to separate concerns for shard discovery
> --------------------------------------------------------------------
>
>                 Key: FLINK-6503
>                 URL: https://issues.apache.org/jira/browse/FLINK-6503
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kinesis Connector
>            Reporter: Tzu-Li (Gordon) Tai
>
> Currently, shard discovery is done within the {{KinesisDataFetcher}} 's loop. I propose
to extract that from the fetcher and encapsulate that in a separate class to separate concerns.
> As can be seen in https://github.com/apache/flink/pull/3001/files#diff-0b702244caed9a73b7b3e1dc8a9cbaebR461,
one downside if we do not do this refactor is apparent when adding rescalability of the Kinesis
consumer. That required shards lookup operations prioir to running the fetcher, which resulted
in the need to expose shard fetching logic external to the `KinesisDataFetcher`. This should
be properly done be separating concerns of shard discovery from the fetcher.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message