flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: keyBy using custom partitioner
Date Wed, 09 Mar 2016 11:57:26 GMT
Hi!

You can currently not override the hash function used by "keyBy()". The
reason is that this function is used in multiple places, for the stream
partitioning, and also for the partitioning of state. Both have to be
aligned.

What you can do is use "partitionCustom(...)" to use an arbitrary
partitioner. However, you cannot window or access state using that...

If you want to partition in a particular way and use windows after that,
you would currently have to do something like a a map function that
generates a special key, and then use keyBy() on that.

Greetings,
Stephan


On Wed, Mar 9, 2016 at 10:07 AM, madhu phatak <phatak.dev@gmail.com> wrote:

> Hi,
> How to use a custom partitioner in keyBy operation? As of now it's using
> hash partitioner to load balance across parallel tasks. I tried custom
> partitioning the schema before calling keyBy operation. It doesn't seem to
> preserve that partition.
>
> --
> Regards,
> Madhukara Phatak
> http://datamantra.io/
>

Mime
View raw message