flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzuli...@apache.org
Subject [1/2] flink git commit: [FLINK-4523] [kinesis] Add documentation for start position configuration
Date Tue, 24 Jan 2017 07:22:49 GMT
Repository: flink
Updated Branches:
  refs/heads/master a8e85a2d5 -> 00d1ad86a

[FLINK-4523] [kinesis] Add documentation for start position configuration

This closes #2916.

Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/00d1ad86
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/00d1ad86
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/00d1ad86

Branch: refs/heads/master
Commit: 00d1ad86a021911f25b0a0aa6e095267d51af1f4
Parents: 8d8a5ab
Author: Tzu-Li (Gordon) Tai <tzulitai@apache.org>
Authored: Mon Jan 9 18:04:12 2017 +0800
Committer: Tzu-Li (Gordon) Tai <tzulitai@apache.org>
Committed: Tue Jan 24 14:20:07 2017 +0800

 docs/dev/connectors/kinesis.md | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/docs/dev/connectors/kinesis.md b/docs/dev/connectors/kinesis.md
index d95fe21..59f3d61 100644
--- a/docs/dev/connectors/kinesis.md
+++ b/docs/dev/connectors/kinesis.md
@@ -113,6 +113,18 @@ configured to enable checkpointing, so that the new shards due to resharding
 Kinesis consumer after the job is restored. This is a temporary limitation that will be resolved
in future versions.
 Please see [FLINK-4341](https://issues.apache.org/jira/browse/FLINK-4341) for more detail.
+#### Configuring Starting Position
+The Flink Kinesis Consumer currently provides the following options to configure where to
start reading Kinesis streams, simply by setting `ConsumerConfigConstants.STREAM_INITIAL_POSITION`
+one of the following values in the provided configuration properties (the naming of the options
identically follows [the namings used by the AWS Kinesis Streams service](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html#API_GetShardIterator_RequestSyntax)):
+- `LATEST`: read all shards of all streams starting from the latest record.
+- `TRIM_HORIZON`: read all shards of all streams starting from the earliest record possible
(data may be trimmed by Kinesis depending on the retention settings).
+- `AT_TIMESTAMP`: read all shards of all streams starting from a specified timestamp. The
timestamp must also be specified in the configuration
+properties by providing a value for `ConsumerConfigConstants.STREAM_INITIAL_TIMESTAMP`, either
in the date pattern
+`yyyy-MM-dd'T'HH:mm:ss.SSSXXX` (for example, `2016-04-04T19:58:46.480-00:00`), or a non-negative
double value representing the number of seconds
+that has elapsed since the Unix epoch (for example, `1459799926.480`).
 #### Fault Tolerance for Exactly-Once User-Defined State Update Semantics
 With Flink's checkpointing enabled, the Flink Kinesis Consumer will consume records from
shards in Kinesis streams and

View raw message