Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C35A3200CD8 for ; Wed, 2 Aug 2017 16:28:42 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C180E1698B0; Wed, 2 Aug 2017 14:28:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DFB5D1698AE for ; Wed, 2 Aug 2017 16:28:41 +0200 (CEST) Received: (qmail 87365 invoked by uid 500); 2 Aug 2017 14:28:36 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 87355 invoked by uid 99); 2 Aug 2017 14:28:36 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Aug 2017 14:28:36 +0000 Received: from Tzu-Lis-MBP.mail (host-58-114-155-242.dynamic.kbtelecom.net [58.114.155.242]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id B9B191A02C0; Wed, 2 Aug 2017 14:28:34 +0000 (UTC) Date: Wed, 2 Aug 2017 22:28:21 +0800 From: "Tzu-Li (Gordon) Tai" To: user@flink.apache.org, aitozi Message-ID: In-Reply-To: <1501681883697-14636.post@n4.nabble.com> References: <1501681883697-14636.post@n4.nabble.com> Subject: Re: KafkaConsumerBase X-Mailer: Airmail (442) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="5981e185_671f495a_1dee" archived-at: Wed, 02 Aug 2017 14:28:42 -0000 --5981e185_671f495a_1dee Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi=21 method shown in=C2=A0KafkaConsumerBase.java (version 1.2.0)=C2=A0 A lot has changed in the =46linkKafkaConsumerBase since version 1.2.0. And if I remember correctly, the =60assignPartitions=60 method was actual= ly a no longer relevant method used in the code, and was properly removed= afterwards. The method for partition assigning in 1.2.0 is called =60assignTopicParti= tions=60, and is used in the open() method. consumerCallBridge.assignPartitions(consumer,=C2=A0 convertKafkaPartitions(subscribedPartitions));=C2=A0 i think here subscribedPartitions is all the partitions , not=C2=A0 subtaskPartitions. This code snippet is from =60KafkaConsumerThread=60, correct=3F As stated above, the partitions are still filtered out to only be the par= titions for each local subtask, using the =60assignTopicPartitions=60 met= hod. So here, the =60subscribedPartitions=60 is not the complete list of = partitions, only the partitions that the subtask should subscribe to. Cheers, Gordon On 2 August 2017 at 9:52:03 PM, aitozi (gjying1314=40gmail.com) wrote: Hi, =20 i have a question that , when we use KafkaConsumerBase, we will have to =20 fetch data from different partition =20 in different parllel thread like the method shown in =20 KafkaConsumerBase.java (version 1.2.0) =20 protected static List assignPartitions( =20 List allPartitions, =20 int numConsumers, int consumerIndex) =7B =20 final List thisSubtaskPartitions =3D new ArrayList<>= ( =20 allPartitions.size() / numConsumers + 1); =20 for (int i =3D 0; i < allPartitions.size(); i++) =7B =20 if (i % numConsumers =3D=3D consumerIndex) =7B =20 thisSubtaskPartitions.add(allPartitions.get(i)); =20 =7D =20 =7D =20 return thisSubtaskPartitions; =20 =7D =20 but i have not find any place invoke this method , in =20 KafkaConsumerThread.java it used =20 consumerCallBridge.assignPartitions(consumer, =20 convertKafkaPartitions(subscribedPartitions)); =20 i think here subscribedPartitions is all the partitions , not =20 subtaskPartitions. Can any one address my problem =20 -- =20 View this message in context: http://apache-flink-user-mailing-list-archi= ve.2336050.n4.nabble.com/KafkaConsumerBase-tp14636.html =20 Sent from the Apache =46link User Mailing List archive. mailing list arch= ive at Nabble.com. =20 --5981e185_671f495a_1dee Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline