flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: A way to control redistribution of operator state?
Date Tue, 14 Feb 2017 07:31:14 GMT
Hi Dmitry,

Technically, from the looks of the internal code around `OperatorStateRepartitioner`, I think
it is certainly possible to be pluggable.
Right now it is just hard coded to use a round-robin repartitioner implementation as default.

However, I’m not sure of the plans in exposing this to the user and making it configurable.
Looping in Stefan (in cc) who mostly worked on this part and see if he can provide more info.

- Gordon

On February 14, 2017 at 2:30:27 AM, Dmitry Golubets (dgolubets@gmail.com) wrote:


It looks impossible to implement a keyed state with operator state now.

I know it sounds like "just use a keyed state", but latter requires updating it on every value
change as opposed to operator state and thus can be expensive (especially if you have to deal
with mutable structures inside which have to be serialized).

The problem is that there is no way to tell Flink how to reassign savepoint parts between
partitions, and thus impossible to route data to correct partitions.

Is there anything I missed or maybe a plan to implement it in future?

Best regards,
View raw message