Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Mon, 10 Apr 2017 06:57:06 -0700
From: "Tzu-Li (Gordon) Tai" <tzulitai@apache.org>
To: dev@flink.apache.org
Message-ID: <etPan.58eb8f32.35af8542.800c@apache.org>
In-Reply-To: <CA+faj9znj3Xg=D-0znqwqa+=3Lxuzo-2HYNt44AQ_rZEMGTNZg@mail.gmail.com>
References: <CA+faj9zPRqWS9KRS2T6+AE69sBWUj0kUb1sKQ8QCF0zyAKVitg@mail.gmail.com>
 <etPan.58eb8b00.b566045.800c@apache.org>
 <CA+faj9znj3Xg=D-0znqwqa+=3Lxuzo-2HYNt44AQ_rZEMGTNZg@mail.gmail.com>
Subject: Re: Possible bug in Kafka producer partitioning logic
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="58eb8f32_1136a914_800c"
archived-at: Mon, 10 Apr 2017 13:57:09 -0000

--58eb8f32_1136a914_800c
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

I would prefer to make this a blocker for a future bugfix actually, and n=
ot 1.2.1.

The reason is that to fix this properly we might need to look again into =
(and possibly change) how partitioners are provided.
The main problem is that the =60open=60 method can only possibly be calle=
d once with the partitions of one topic.
So, we might need the user to provide multiple partitioners, one for each=
 of all the possible topics that will be written to.

One way or another, my gut feeling is that this would need somewhat sligh=
t change to the Kafka producer APIs.
And I=E2=80=99m not so sure of rushing API changes into releases.


On April 10, 2017 at 6:46:29 AM, Gyula =46=C3=B3ra (gyula.fora=40gmail.co=
m) wrote:

Thanks for checking this out. =20

I would say this is definitely a blocking issue for the bugfix release, =20
what do you think=3F =20

Gyula =20

Tzu-Li (Gordon) Tai <tzulitai=40apache.org> ezt =C3=ADrta (id=C5=91pont: =
2017. =C3=A1pr. =20
10., H, 15:39): =20

Hi Gyula, =20

Yes, I think the semantics of the Partitioner interface is a bit off. =20
The =60numPartitions=60 value ideally should be the number of partitions =
of the =20
=60targetTopic=60. =20

Here=E2=80=99s a JIRA I just filed to track the issue: =20
https://issues.apache.org/jira/browse/=46LINK-6288. =20

Cheers, =20
Gordon =20

On April 10, 2017 at 1:16:18 AM, Gyula =46=C3=B3ra (gyula.fora=40gmail.co=
m) wrote: =20

Hi all, =20

We had some problems with custom partitioning for the 0.8 Kafka producer =
=20
and now that I checked the code it seems there might be a problem with th=
e =20
logic. =20

The producer determines the number of partitions in the open method and =20
seems to be using that as a value passed to the custom partitioner for =20
producing the records. =20
This will however only work if the defaultTopicId (topic) has the same =20
number of partitions as all other topics in the kafka cluster when =20
producing to multiple topics. =20

In our case the default topic had 16 and new ones have 3 as default so it=
 =20
gives an out of range partition error. =20

Is my understanding correct or am I overlooking something=3F =20

Thank you=21 =20
Gyula =20

--58eb8f32_1136a914_800c--