flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Wei <tony19920...@gmail.com>
Subject Re: dynamically partitioned stream
Date Thu, 31 Aug 2017 08:59:19 GMT
Hi Martin,

Let me understand your question first.
You have two Stream: Data Stream and Control Stream and you want to select
data in Data Stream based on the key set got from Control Stream.

If I were not misunderstanding your question, I think SideInput is what you
want.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API#FLIP-17SideInputsforDataStreamAPI-StoringSide-InputData
It lets you to define one stream as a SideInput and can be assigned to the
other stream, then the data in SideInput stream will be broadcasted.

So far, I have no idea if there is any solution to solve this without
SideInput.

Best,
Tony Wei

2017-08-31 16:10 GMT+08:00 Martin Eden <martineden131@gmail.com>:

> Hi all,
>
> I am trying to implement the following using Flink:
>
> I have 2 input message streams:
>
> 1. Data Stream:
> KEY VALUE TIME
> .
> .
> .
> C      V6        6
> B      V6        6
> A      V5        5
> A      V4        4
> C      V3        3
> A      V3        3
> B      V3        3
> B      V2        2
> A      V1        1
>
> 2. Control Stream:
> Lambda  ArgumentKeys TIME
> .
> .
> .
> f2            [A, C]                 4
> f1            [A, B, C]            1
>
> I want to apply the lambdas coming in the control stream to the selection
> of keys that are coming in the data stream.
>
> Since we have 2 streams I naturally thought of connecting them using
> .connect. For this I need to key both of them by a certain criteria. And
> here lies the problem, how can I make sure the messages with keys A,B,C
> specified in the control stream end up in the same task as well as the
> control message (f1, [A, B, C]) itself. Basically I don't know how to key
> by to achieve this.
>
> I suspect a custom partitioner is required that partitions the data stream
> based on the messages in the control stream? Is this even possible?
>
> Any suggestions welcomed!
>
> Thanks,
> M
>
>
>
>
>
>
>

Mime
View raw message