Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
Sender: ewenstephan@gmail.com
In-Reply-To: 
 <CANpNmWXEbiwrN8pCuVjZyzp_+YGTgdqDBFWZcqeaLtkSZxVdZg@mail.gmail.com>
References: 
 <CANpNmWXEbiwrN8pCuVjZyzp_+YGTgdqDBFWZcqeaLtkSZxVdZg@mail.gmail.com>
Date: Wed, 9 Mar 2016 12:57:26 +0100
Message-ID: 
 <CANC1h_vJ8gmFsCFLO3ovqAEtmnot5nLcMuQ7hVf2Tv85yQqtyw@mail.gmail.com>
Subject: Re: keyBy using custom partitioner
From: Stephan Ewen <sewen@apache.org>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0810acd34b0b052d9c6666

--94eb2c0810acd34b0b052d9c6666
Content-Type: text/plain; charset=UTF-8

Hi!

You can currently not override the hash function used by "keyBy()". The
reason is that this function is used in multiple places, for the stream
partitioning, and also for the partitioning of state. Both have to be
aligned.

What you can do is use "partitionCustom(...)" to use an arbitrary
partitioner. However, you cannot window or access state using that...

If you want to partition in a particular way and use windows after that,
you would currently have to do something like a a map function that
generates a special key, and then use keyBy() on that.

Greetings,
Stephan


On Wed, Mar 9, 2016 at 10:07 AM, madhu phatak <phatak.dev@gmail.com> wrote:

> Hi,
> How to use a custom partitioner in keyBy operation? As of now it's using
> hash partitioner to load balance across parallel tasks. I tried custom
> partitioning the schema before calling keyBy operation. It doesn't seem to
> preserve that partition.
>
> --
> Regards,
> Madhukara Phatak
> http://datamantra.io/
>

--94eb2c0810acd34b0b052d9c6666
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi!<div><br></div><div>You can currently not override the =
hash function used by &quot;keyBy()&quot;. The reason is that this function=
 is used in multiple places, for the stream partitioning, and also for the =
partitioning of state. Both have to be aligned.</div><div><br></div><div>Wh=
at you can do is use &quot;partitionCustom(...)&quot; to use an arbitrary p=
artitioner. However, you cannot window or access state using that...</div><=
div><br></div><div>If you want to partition in a particular way and use win=
dows after that, you would currently have to do something like a a map func=
tion that generates a special key, and then use keyBy() on that.</div><div>=
<br></div><div>Greetings,</div><div>Stephan</div><div><br></div></div><div =
class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Mar 9, 2016 at=
 10:07 AM, madhu phatak <span dir=3D"ltr">&lt;<a href=3D"mailto:phatak.dev@=
gmail.com" target=3D"_blank">phatak.dev@gmail.com</a>&gt;</span> wrote:<br>=
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi,<div>How to use a custom=
 partitioner in keyBy operation? As of now it&#39;s using hash partitioner =
to load balance across parallel tasks. I tried custom partitioning the sche=
ma before calling keyBy operation. It doesn&#39;t seem to preserve that par=
tition.=C2=A0</div><span class=3D"HOEnZb"><font color=3D"#888888"><div><div=
><br></div>-- <br><div><div dir=3D"ltr"><div style=3D"font-size:12.80000019=
07349px">Regards,<br></div><div style=3D"font-size:12.8000001907349px">Madh=
ukara Phatak<br><a href=3D"http://datamantra.io/" target=3D"_blank">http://=
datamantra.io/</a></div></div></div>
</div></font></span></div>
</blockquote></div><br></div>

--94eb2c0810acd34b0b052d9c6666--