flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rmetz...@apache.org
Subject flink git commit: [hotfix][docs] Add note about Kinesis producer limitations
Date Tue, 12 Jul 2016 11:58:29 GMT
Repository: flink
Updated Branches:
  refs/heads/master f0387aca4 -> 971dcc5de


[hotfix][docs] Add note about Kinesis producer limitations

This closes #2229


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/971dcc5d
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/971dcc5d
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/971dcc5d

Branch: refs/heads/master
Commit: 971dcc5dec802bea7f258781da111321167fb0e5
Parents: f0387ac
Author: Robert Metzger <rmetzger@apache.org>
Authored: Tue Jul 12 11:18:42 2016 +0200
Committer: Robert Metzger <rmetzger@apache.org>
Committed: Tue Jul 12 13:57:55 2016 +0200

----------------------------------------------------------------------
 docs/apis/streaming/connectors/kinesis.md | 11 ++++++-----
 docs/apis/streaming/fault_tolerance.md    |  9 +++++----
 2 files changed, 11 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/971dcc5d/docs/apis/streaming/connectors/kinesis.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/connectors/kinesis.md b/docs/apis/streaming/connectors/kinesis.md
index 43a87f9..a014326 100644
--- a/docs/apis/streaming/connectors/kinesis.md
+++ b/docs/apis/streaming/connectors/kinesis.md
@@ -50,9 +50,8 @@ mvn clean install -Pinclude-kinesis -DskipTests
 {% endhighlight %}
 
 
-
-Note that the streaming connectors are not part of the binary distribution. 
-See how to link with them for cluster execution [here]({{site.baseurl}}/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+The streaming connectors are not part of the binary distribution. See how to link with them
for cluster 
+execution [here]({{site.baseurl}}/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
 
 ### Using the Amazon Kinesis Streams Service
 Follow the instructions from the [Amazon Kinesis Streams Developer Guide](https://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-create-stream.html)
@@ -240,8 +239,10 @@ consumer when calling this API can also be modified by using the other
keys pref
 ### Kinesis Producer
 
 The `FlinkKinesisProducer` is used for putting data from a Flink stream into a Kinesis stream.
Note that the producer is not participating in
-Flink's checkpointing and doesn't provide exactly-once processing guarantees. In case of
a failure, data will be written again
-to Kinesis, leading to duplicates. This behavior is usually called "at-least-once" semantics.
+Flink's checkpointing and doesn't provide exactly-once processing guarantees. 
+Also, the Kinesis producer does not guarantee that records are written in order to the shards
(See [here](https://github.com/awslabs/amazon-kinesis-producer/issues/23) and [here](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#API_PutRecord_RequestSyntax)
for more details). 
+
+In case of a failure or a resharding, data will be written again to Kinesis, leading to duplicates.
This behavior is usually called "at-least-once" semantics.
 
 To put data into a Kinesis stream, make sure the stream is marked as "ACTIVE" in the AWS
dashboard.
 

http://git-wip-us.apache.org/repos/asf/flink/blob/971dcc5d/docs/apis/streaming/fault_tolerance.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/fault_tolerance.md b/docs/apis/streaming/fault_tolerance.md
index 80b25ef..99221e5 100644
--- a/docs/apis/streaming/fault_tolerance.md
+++ b/docs/apis/streaming/fault_tolerance.md
@@ -103,8 +103,9 @@ env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
 ### Fault Tolerance Guarantees of Data Sources and Sinks
 
 Flink can guarantee exactly-once state updates to user-defined state only when the source
participates in the
-snapshotting mechanism. This is currently guaranteed for the Kafka source and AWS Kinesis
Streams source (and internal number generators), but
-not for other sources. The following table lists the state update guarantees of Flink coupled
with the bundled sources:
+snapshotting mechanism. The following table lists the state update guarantees of Flink coupled
with the bundled connectors.
+
+Please read the documentation of each connector to understand the details of the fault tolerance
guarantees.
 
 <table class="table table-bordered">
   <thead>
@@ -142,8 +143,8 @@ not for other sources. The following table lists the state update guarantees
of
         </tr>
         <tr>
             <td>Files</td>
-            <td>at least once</td>
-            <td>At failure the file will be read from the beginning</td>
+            <td>exactly once</td>
+            <td></td>
         </tr>
         <tr>
             <td>Sockets</td>


Mime
View raw message