flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Send events to parallel operator instances
Date Wed, 03 Jun 2015 10:20:21 GMT
Ah ok. And by distributed you mean that the element that starts the
window can be processed on a different machine than the element that
finishes the window?

On Wed, Jun 3, 2015 at 12:11 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:
> This is not connected to the current implementation. So lets not talk about
> that.
>
> This is about theoretical limits to support distributed delta policies
> which has far reaching implications for the windowing policies one can
> implement in a prallel way.
>
> But you are welcome to throw in any constructive ideas :)
> On Wed, Jun 3, 2015 at 11:49 AM Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>
>> Part of the reason for my question is this:
>> https://issues.apache.org/jira/browse/FLINK-1967. Especially my latest
>> comment there. If we want this, I think we have to overhaul the
>> windowing system anyways and then it doesn't make sense to explore
>> complicated workarounds for the current system.
>>
>> On Wed, Jun 3, 2015 at 11:07 AM, Gyula Fóra <gyula.fora@gmail.com> wrote:
>> > There are simple ways of implementing it in a non-distributed or
>> > inconsistent fashion.
>> > On Wed, Jun 3, 2015 at 8:55 AM Aljoscha Krettek <aljoscha@apache.org>
>> wrote:
>> >
>> >> This already sounds awfully complicated. Is there no other way to
>> >> implement the delta windows?
>> >>
>> >> On Wed, Jun 3, 2015 at 7:52 AM, Gyula Fóra <gyula.fora@gmail.com>
>> wrote:
>> >> > Hi Ufuk,
>> >> >
>> >> > In the concrete use case I have in mind I only want to send events
to
>> >> > another subtask of the same task vertex.
>> >> >
>> >> > Specifically: if we want to do distributed delta based windows we
>> need to
>> >> > send after every trigger the element that has triggered the current
>> >> window.
>> >> > So practically I want to broadcast some event regularly to all
>> subtasks
>> >> of
>> >> > the same operator.
>> >> >
>> >> > In this case the operators would wait until they receive this event
>> so we
>> >> > need to make sure that this event sending is not blocked by the actual
>> >> > records.
>> >> >
>> >> > Gyula
>> >> >
>> >> > On Tuesday, June 2, 2015, Ufuk Celebi <uce@apache.org> wrote:
>> >> >
>> >> >>
>> >> >> On 02 Jun 2015, at 22:45, Gyula Fóra <gyfora@apache.org
>> <javascript:;>>
>> >> >> wrote:
>> >> >> > I am wondering, what is the suggested way to send some events
>> >> directly to
>> >> >> > another parallel instance in a flink job? For example from
one
>> mapper
>> >> to
>> >> >> > another mapper (of the same operator).
>> >> >> >
>> >> >> > Do we have any internal support for this? The first thing
that we
>> >> thought
>> >> >> > of is iterations but that is clearly an overkill.
>> >> >>
>> >> >> There is no support for this at the moment. Any parallel instance?
>> Or a
>> >> >> subtask instance of the same task?
>> >> >>
>> >> >> Can you provide more input on the use case? It is certainly possible
>> to
>> >> >> add support for this.
>> >> >>
>> >> >> If the events don't need to be inline with the records, we can
easily
>> >> >> setup the TaskEventDispatcher as a separate actor (or extend the
task
>> >> >> manager) to process both backwards flowing events and in general
any
>> >> events
>> >> >> that don't need to be inline with the records. The task deployment
>> >> >> descriptors need to be extended with the extra parallel instance
>> >> >> information.
>> >> >>
>> >> >> – Ufuk
>> >>
>>

Mime
View raw message