kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From guozh...@apache.org
Subject kafka git commit: MINOR: update stream docs for kip-134
Date Tue, 06 Jun 2017 16:37:33 GMT
Repository: kafka
Updated Branches:
  refs/heads/0.11.0 c39fea37e -> 503f92721


MINOR: update stream docs for kip-134

Add a section in the streams docs about the broker config introduced in KIP-134

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Michael G. Noll, Bill Bejeck, Guozhang Wang

Closes #3179 from dguy/kip134-doc

(cherry picked from commit ad8e7cf5e82386f8596025b06fab6b1cbd0b9109)
Signed-off-by: Guozhang Wang <wangguoz@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/503f9272
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/503f9272
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/503f9272

Branch: refs/heads/0.11.0
Commit: 503f927210a91ffd6fa90703099276083876f02d
Parents: c39fea3
Author: Damian Guy <damian.guy@gmail.com>
Authored: Tue Jun 6 09:37:13 2017 -0700
Committer: Guozhang Wang <wangguoz@gmail.com>
Committed: Tue Jun 6 09:37:29 2017 -0700

----------------------------------------------------------------------
 docs/streams.html | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kafka/blob/503f9272/docs/streams.html
----------------------------------------------------------------------
diff --git a/docs/streams.html b/docs/streams.html
index d151632..bff9bf0 100644
--- a/docs/streams.html
+++ b/docs/streams.html
@@ -658,7 +658,7 @@ KStream&lt;String, String&gt; materialized = joined.through("topic4");
 </pre>
         <br>
 
-        <h3><a id="streams_execute" href="#streams_execute">Application Configuration
and Execution</a></h3>
+        <h3><a id="streams_configure_execute" href="#streams_configure_execute">Application
Configuration and Execution</a></h3>
 
         <p>
         Besides defining the topology, developers will also need to configure their applications
@@ -693,6 +693,7 @@ settings.put(... , ...);
 StreamsConfig config = new StreamsConfig(settings);
 </pre>
 
+        <h4><a id="streams_client_config" href="#streams_clients_config">Producer
and Consumer Configuration</a></h4>
         <p>
         Apart from Kafka Streams' own configuration parameters you can also specify parameters
for the Kafka consumers and producers that are used internally,
         depending on the needs of your application. Similar to the Streams settings you define
any such consumer and/or producer settings via <code>StreamsConfig</code>.
@@ -720,6 +721,24 @@ settings.put(StreamsConfig.consumerPrefix(ConsumerConfig.RECEIVE_BUFFER_CONFIG),
 settings.put(StremasConfig.producerConfig(ProducerConfig.RECEIVE_BUFFER_CONFIG), 64 * 1024);
 </pre>
 
+        <h4><a id="streams_broker_config" href="#streams_broker_config">Broker
Configuration</a></h4>
+        <p>
+            Introduced in 0.11.0 is a new broker config that is particularly relevant to
Kafka Streams applications, <code>group.initial.rebalance.delay.ms</code>.
+            This config specifies the time, in milliseconds, that the <code>GroupCoordinator</code>
will delay the initial consumer rebalance.
+            The rebalance will be further delayed by the value of <code>group.initial.rebalance.delay.ms</code>
as each new member joins the consumer group, up to a maximum of the value set by <code>max.poll.interval.ms</code>.
+            The net benefit is that this should reduce the overall startup time for Kafka
Streams applications with more than one thread.
+            The default value for <code>group.initial.rebalance.delay.ms</code>
is 3 seconds.
+        </p>
+        <p>
+            In practice this means that if you are starting up your Kafka Streams app from
a cold start, then when the first member joins the group there will be at least a 3 second
delay before it is assigned any tasks.
+            If any other members join the group within the initial 3 seconds, then there
will be a further 3 second delay.
+            Once no new members have joined the group within the 3 second delay, or <code>max.poll.interval.ms</code>
is reached, then the group rebalance can complete and all current members will be assigned
tasks.
+            The benefit of this approach, particularly for Kafka Streams applications, is
that we can now delay the assignment and re-assignment of potentially expensive tasks as new
members join.
+            So we can avoid the situation where one instance is assigned all tasks, begins
restoring/processing, only to shortly after be rebalanced, and then have to start again with
half of the tasks and so on.
+        </p>
+
+
+        <h4><a id="streams_execute" href="#streams_execute">Executing Your Kafka
Streams Application</a></h4>
         <p>
         You can call Kafka Streams from anywhere in your application code.
         Very commonly though you would do so within the <code>main()</code> method
of your application, or some variant thereof.


Mime
View raw message