Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANC1h_uowG08DRc_WwV0+aavrPWuwuxvJYufXj=PetnmMSMmXg@mail.gmail.com>
References: 
 <CAFDmHdMoqVsoKUyw2NkyoQzaA-qJ39PWpkyb7GbgwZ4vTJvKzg@mail.gmail.com>
 <CANC1h_spLAe_Wk9pyii3q7MKksAdBMpcW_Hvv1dQiy4sA_6veQ@mail.gmail.com>
 <CANpG74EnAUVVbnnfgtgZDTQ7b_cophErqCPTYhEiyyDFPtb8jQ@mail.gmail.com>
 <CANC1h_uowG08DRc_WwV0+aavrPWuwuxvJYufXj=PetnmMSMmXg@mail.gmail.com>
From: Elias Levy <fearsome.lucidity@gmail.com>
Date: Wed, 13 Apr 2016 18:10:48 -0700
Message-ID: 
 <CAFDmHdOV_caGnzSZVMU5_BuYWayOM=80iGyD_g0dncQ=g+J6bw@mail.gmail.com>
Subject: Re: Does Kafka connector leverage Kafka message keys?
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1134d33cb8d07d05306791e8

--001a1134d33cb8d07d05306791e8
Content-Type: text/plain; charset=UTF-8

On Wed, Apr 13, 2016 at 2:10 AM, Stephan Ewen <sewen@apache.org> wrote:

> If you want to use Flink's internal key/value state, however, you need to
> let Flink re-partition the data by using "keyBy()". That is because Flink's
> internal sharding of state (including the re-sharding to adjust parallelism
> we are currently working on) follows a dedicated hashing scheme which is
> with all likelihood different from the partition function that writes the
> key/value pairs to the Kafka Topics.
>

That is interesting, if somewhat disappointing.  I was hoping that
performing a keyBy from a Kafka source would perform no reshuffling if you
used the same value as you used for the Kafka message key.  But it makes
sense if you are using different hash functions.

It may be useful to have a variant of keyBy() that converts the stream to a
KeyedStream but performs no shuffling if the caller is certain that the
DataStream is already partitioned by the given key.

--001a1134d33cb8d07d05306791e8
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">On Wed, Apr 13, 2016 at 2:10 AM, Stephan Ewen <span dir=3D=
"ltr">&lt;<a href=3D"mailto:sewen@apache.org" target=3D"_blank">sewen@apach=
e.org</a>&gt;</span> wrote:<br><div class=3D"gmail_extra"><div class=3D"gma=
il_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>If you want =
to use Flink&#39;s internal key/value state, however, you need to let Flink=
 re-partition the data by using &quot;keyBy()&quot;. That is because Flink&=
#39;s internal sharding of state (including the re-sharding to adjust paral=
lelism we are currently working on) follows a dedicated hashing scheme whic=
h is with all likelihood different from the partition function that writes =
the key/value pairs to the Kafka Topics.<br></div></div></blockquote><div><=
br></div><div>That is interesting, if somewhat disappointing.=C2=A0 I was h=
oping that performing a keyBy from a Kafka source would perform no reshuffl=
ing if you used the same value as you used for the Kafka message key.=C2=A0=
 But it makes sense if you are using different hash functions.</div><div><b=
r></div><div>It may be useful to have a variant of keyBy() that converts th=
e stream to a KeyedStream but performs no shuffling if the caller is certai=
n that the DataStream is already partitioned by the given key.</div><div><b=
r></div></div><br></div></div>

--001a1134d33cb8d07d05306791e8--