Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E6AFF200C3F for ; Wed, 8 Mar 2017 05:07:42 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E5356160B82; Wed, 8 Mar 2017 04:07:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3C0D3160B68 for ; Wed, 8 Mar 2017 05:07:42 +0100 (CET) Received: (qmail 15395 invoked by uid 500); 8 Mar 2017 04:07:41 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 15376 invoked by uid 99); 8 Mar 2017 04:07:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Mar 2017 04:07:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id DBADE1AA982 for ; Wed, 8 Mar 2017 04:07:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.651 X-Spam-Level: X-Spam-Status: No, score=0.651 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id QK98PEVXfr0w for ; Wed, 8 Mar 2017 04:07:40 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id A18F75FBE2 for ; Wed, 8 Mar 2017 04:07:39 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B9452E05B1 for ; Wed, 8 Mar 2017 04:07:38 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0F1782416C for ; Wed, 8 Mar 2017 04:07:38 +0000 (UTC) Date: Wed, 8 Mar 2017 04:07:38 +0000 (UTC) From: "LvDongrong (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (SPARK-19863) Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 08 Mar 2017 04:07:43 -0000 LvDongrong created SPARK-19863: ---------------------------------- Summary: Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application Key: SPARK-19863 URL: https://issues.apache.org/jira/browse/SPARK-19863 Project: Spark Issue Type: Bug Components: DStreams, Input/Output Affects Versions: 2.1.0 Reporter: LvDongrong Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application. In Spark 2.x, the kafka consumer was replaced by CachedKafkaConsumer (some KafkaConsumer will keep establishing the kafka cluster), and cannot change the way. In fact ,The KafkaRDD(used by DirectKafkaInputDStream to connect kafka) provide the parameter useConsumerCache to choose Whether to use the CachedKafkaConsumer, but the DirectKafkaInputDStream set the parameter true. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org