flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6141) Add buffering service for stream operators
Date Thu, 06 Apr 2017 07:06:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958428#comment-15958428
] 

Aljoscha Krettek commented on FLINK-6141:
-----------------------------------------

[~ventura] Your ideas definitely don't sound off. Currently I'm very busy with Flink Forward
(http://sf.flink-forward.org), though. I'll get back to this once I have time again.

> Add buffering service for stream operators
> ------------------------------------------
>
>                 Key: FLINK-6141
>                 URL: https://issues.apache.org/jira/browse/FLINK-6141
>             Project: Flink
>          Issue Type: Sub-task
>          Components: DataStream API
>            Reporter: Aljoscha Krettek
>
> As mentioned in https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
we need a way of buffering incoming elements until a side input that is required for processing
them is ready.
> There has to be an implementation for non-keyed operators and for keyed operators because
in keyed operators we need to ensure that we store the buffered elements in the correct key
group when checkpointing.
> For the interface, I propose this:
> {code}
> @PublicEvolving
> public interface ElementBuffer<T, N> {
> 	/**
> 	 * Adds the given element to the buffer for the given namespace.
> 	 */
> 	void add(N namespace, T element);
> 	/**
> 	 * Returns an {@code Iterable} over all buffered elements for the given namespace.
> 	 */
> 	Iterable<T> values(N namespace);
> 	/**
> 	 * Clears all buffered elements for the given namespace. 
> 	 */
> 	void clear(N namespace);
> }
> {code}
> {{AbstractStreamOperator}} would provide a method {{getElementBuffer()}} that would return
the appropriate implementation for a non-keyed or keyed operator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message