flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Dataset and select/split functionality
Date Fri, 03 Mar 2017 10:52:56 GMT
Hi CPC,

we had several requests in the past to add this features. However, adding
select/split for DataSet is much! more work than you would expect.
As you pointed out, we have to go through the optimizer, which assumes that
the outputs of a function are replicated.
This is pretty much wired in and you would have to touch a lot of code.

I'm sorry, but am not comfortable doing such a big change.
IMO, the potential gains are not worth the effort of implementation and
verification and the risk of breaking something.

Best, Fabian



2017-03-02 16:31 GMT+01:00 CPC <achalil@gmail.com>:

> Hi all,
>
> We will try to implement select/split functionality for batch api. We
> looked at streaming side and understand how it works but since streaming
> side does not include an optimizer it was easier. Since adding such a
> runtime operator will affect optimizer layer as well, is there a part that
> you want us to pay particular attention to?
>
> Thanks...
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message