spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LvDongrong (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-19863) Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application
Date Wed, 08 Mar 2017 04:07:38 GMT
LvDongrong created SPARK-19863:
----------------------------------

             Summary: Whether or not use CachedKafkaConsumer need to be configured, when you
use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application
                 Key: SPARK-19863
                 URL: https://issues.apache.org/jira/browse/SPARK-19863
             Project: Spark
          Issue Type: Bug
          Components: DStreams, Input/Output
    Affects Versions: 2.1.0
            Reporter: LvDongrong


Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream
to connect the kafka in a Spark Streaming application. In Spark 2.x, the kafka consumer was
replaced by CachedKafkaConsumer (some KafkaConsumer will keep establishing the kafka cluster),
and cannot change the way. In fact ,The KafkaRDD(used by DirectKafkaInputDStream to connect
kafka) provide the parameter useConsumerCache to choose Whether to use the CachedKafkaConsumer,
but the DirectKafkaInputDStream set the parameter true.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message