spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zsxw...@apache.org
Subject spark git commit: [STREAMING][KAFKA][DOC] clarify kafka settings needed for larger batches
Date Fri, 21 Oct 2016 23:27:22 GMT
Repository: spark
Updated Branches:
  refs/heads/master 268ccb9a4 -> c9720b219


[STREAMING][KAFKA][DOC] clarify kafka settings needed for larger batches

## What changes were proposed in this pull request?

Minor doc change to mention kafka configuration for larger spark batches.

## How was this patch tested?

Doc change only, confirmed via jekyll.

The configuration issue was discussed / confirmed with users on the mailing list.

Author: cody koeninger <cody@koeninger.org>

Closes #15570 from koeninger/kafka-doc-heartbeat.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c9720b21
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c9720b21
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c9720b21

Branch: refs/heads/master
Commit: c9720b2195a465653690b3e221ce789142217b0d
Parents: 268ccb9
Author: cody koeninger <cody@koeninger.org>
Authored: Fri Oct 21 16:27:19 2016 -0700
Committer: Shixiong Zhu <shixiong@databricks.com>
Committed: Fri Oct 21 16:27:19 2016 -0700

----------------------------------------------------------------------
 docs/streaming-kafka-0-10-integration.md | 1 +
 1 file changed, 1 insertion(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/c9720b21/docs/streaming-kafka-0-10-integration.md
----------------------------------------------------------------------
diff --git a/docs/streaming-kafka-0-10-integration.md b/docs/streaming-kafka-0-10-integration.md
index 456b845..de95ea9 100644
--- a/docs/streaming-kafka-0-10-integration.md
+++ b/docs/streaming-kafka-0-10-integration.md
@@ -48,6 +48,7 @@ Each item in the stream is a [ConsumerRecord](http://kafka.apache.org/0100/javad
 </div>
 
 For possible kafkaParams, see [Kafka consumer config docs](http://kafka.apache.org/documentation.html#newconsumerconfigs).
+If your Spark batch duration is larger than the default Kafka heartbeat session timeout (30
seconds), increase heartbeat.interval.ms and session.timeout.ms appropriately.  For batches
larger than 5 minutes, this will require changing group.max.session.timeout.ms on the broker.
 Note that the example sets enable.auto.commit to false, for discussion see [Storing Offsets](streaming-kafka-0-10-integration.html#storing-offsets)
below.
 
 ### LocationStrategies


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message