flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Send events to parallel operator instances
Date Wed, 03 Jun 2015 09:49:04 GMT
Part of the reason for my question is this:
https://issues.apache.org/jira/browse/FLINK-1967. Especially my latest
comment there. If we want this, I think we have to overhaul the
windowing system anyways and then it doesn't make sense to explore
complicated workarounds for the current system.

On Wed, Jun 3, 2015 at 11:07 AM, Gyula Fóra <gyula.fora@gmail.com> wrote:
> There are simple ways of implementing it in a non-distributed or
> inconsistent fashion.
> On Wed, Jun 3, 2015 at 8:55 AM Aljoscha Krettek <aljoscha@apache.org> wrote:
>
>> This already sounds awfully complicated. Is there no other way to
>> implement the delta windows?
>>
>> On Wed, Jun 3, 2015 at 7:52 AM, Gyula Fóra <gyula.fora@gmail.com> wrote:
>> > Hi Ufuk,
>> >
>> > In the concrete use case I have in mind I only want to send events to
>> > another subtask of the same task vertex.
>> >
>> > Specifically: if we want to do distributed delta based windows we need to
>> > send after every trigger the element that has triggered the current
>> window.
>> > So practically I want to broadcast some event regularly to all subtasks
>> of
>> > the same operator.
>> >
>> > In this case the operators would wait until they receive this event so we
>> > need to make sure that this event sending is not blocked by the actual
>> > records.
>> >
>> > Gyula
>> >
>> > On Tuesday, June 2, 2015, Ufuk Celebi <uce@apache.org> wrote:
>> >
>> >>
>> >> On 02 Jun 2015, at 22:45, Gyula Fóra <gyfora@apache.org <javascript:;>>
>> >> wrote:
>> >> > I am wondering, what is the suggested way to send some events
>> directly to
>> >> > another parallel instance in a flink job? For example from one mapper
>> to
>> >> > another mapper (of the same operator).
>> >> >
>> >> > Do we have any internal support for this? The first thing that we
>> thought
>> >> > of is iterations but that is clearly an overkill.
>> >>
>> >> There is no support for this at the moment. Any parallel instance? Or a
>> >> subtask instance of the same task?
>> >>
>> >> Can you provide more input on the use case? It is certainly possible to
>> >> add support for this.
>> >>
>> >> If the events don't need to be inline with the records, we can easily
>> >> setup the TaskEventDispatcher as a separate actor (or extend the task
>> >> manager) to process both backwards flowing events and in general any
>> events
>> >> that don't need to be inline with the records. The task deployment
>> >> descriptors need to be extended with the extra parallel instance
>> >> information.
>> >>
>> >> – Ufuk
>>

Mime
View raw message