flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Question about Re-Partition Operator
Date Wed, 24 Sep 2014 15:00:09 GMT
(mostly @fabian) why is the re-partition operator special-cased like
it is now? You can only do map/filter on partitioned data. Wouldn't it
be nice if we had a general re-partition operator. Operators that
normally do their own repartitioning would notice that the data is
already partitioned and use that. The way it is now, the
implementation relies heavily on special-case handling.

In the long run we could even introduce combine and sort as special
operators that users could insert themselves. The optimiser would then
also insert these before operations when required. This would
simplify/generalise things a bit.

What do you think?


View raw message