I think this would benefit from a FLIP, that neatly sums up the options, and which then gives us also a point where we can vote and ratify a decision.

As a gut feeling, I most like Option 3). Initially I would have preferred option 1) (because of a sense of API purity), but by now I think it’s good that users have this simpler option.


On 8. Jul 2019, at 06:39, Xingcan Cui <xingcanc@gmail.com> wrote:

Hi all,

Thanks for your participation.

In this thread, we got one +1 for option 1 and option 3, respectively. In the original thread[1], we got two +1 for option 1, one +1 for option 2, and five +1 and one -1 for option 3.

To summarize,

Option 1 (port side output to flatMap and deprecate split/select): three +1
Option 2 (introduce a new split/select and deprecate existing one): one +1
Option 3 ("correct" the existing split/select): six +1 and one -1

It seems that most people involved are in favor of "correcting" the existing split/select. However, this will definitely break the API compatibility, in a subtle way.

IMO, the real behavior of consecutive split/select's has never been thoroughly clarified. Even in the community, it hard to say that we come into a consensus on its real semantics[2-4]. Though the initial design is not ambiguous, there's no doubt that its concept has drifted. 

As the split/select is quite an ancient API, I cc'ed this to more members. It couldn't be better if you can share your opinions on this.


[1] https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E
[2] https://issues.apache.org/jira/browse/FLINK-1772
[3] https://issues.apache.org/jira/browse/FLINK-5031
[4] https://issues.apache.org/jira/browse/FLINK-11084

On Jul 5, 2019, at 12:04 AM, 杨力 <bill.lee.y@gmail.com> wrote:

I prefer the 1) approach. I used to carry fields, which is needed only for splitting, in the outputs of flatMap functions. Replacing it with outputTags would simplify data structures.

Xingcan Cui <xingcanc@gmail.com <mailto:xingcanc@gmail.com>> 于 2019年7月5日周五 上午2:20写道:
Hi folks,

Two weeks ago, I started a thread [1] discussing whether we should discard the split/select methods (which have been marked as deprecation since v1.7) in DataStream API. 

The fact is, these methods will cause "unexpected" results when using consecutively (e.g., ds.split(a).select(b).split(c).select(d)) or multi-times on the same target (e.g., ds.split(a).select(b), ds.split(c).select(d)). The reason is that following the initial design, the new split/select logic will always override the existing one on the same target operator, rather than append to it. Some users may not be aware of that, but if you do, a current solution would be to use the more powerful side output feature [2].

FLINK-11084 <https://issues.apache.org/jira/browse/FLINK-11084> added some restrictions to the existing split/select logic and suggest to replace it with side output in the future. However, considering that the side output is currently only available in the process function layer and the split/select could have been widely used in many real-world applications, we'd like to start a vote andlisten to the community on how to deal with them.

In the discussion thread [1], we proposed three solutions as follows. All of them are feasible but have different impacts on the public API.

1) Port the side output feature to DataStream API's flatMap and replace split/select with it.

2) Introduce a dedicated function in DataStream API (with the "correct" behavior but a different name) that can be used to replace the existing split/select.

3) Keep split/select but change the behavior/semantic to be "correct".

Note that this is just a vote for gathering information, so feel free to participate and share your opinions.

The voting time will end on July 7th 17:00 EDT.


[1] https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E<https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E>
[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/side_output.html <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/side_output.html>