flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifei Li <lee891...@gmail.com>
Subject Re: Does Flink support joining multiple streams based on event time window now?
Date Tue, 19 Apr 2016 16:08:58 GMT
Hi Till and Aljoscha,

Thank you so much for your suggestions and I'll try them out. I have
another question.

Since S2 my be days delayed, so there are may be lots of windows and large
amount of data stored in memory waiting for computation. How does Flink
deal with that´╝č

Thanks,

Yifei

On Tue, Apr 19, 2016 at 2:55 AM, Till Rohrmann <trohrmann@apache.org> wrote:

> Hi Yifei,
>
> if you don't wanna implement your own join operator, then you could also
> chain two join operations. I created a small example to demonstrate that:
> https://gist.github.com/tillrohrmann/c074b4eedb9deaf9c8ca2a5e124800f3.
> However, bare in mind that for this approach you will construct two windows
> which might be a bit more costly than Aljoscha's approach.
>
> Cheers,
> Till
>
> On Tue, Apr 19, 2016 at 11:32 AM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>
>> Hi,
>> right now, there is no built-in support for n-ary joins. I am working on
>> this, however.
>>
>> For now you can simulate n-ary joins by using a tagged union and doing
>> the join yourself in a WindowFunction. I created a small example that
>> demonstrates this:
>> https://gist.github.com/aljoscha/a2a213d90c7c1bc67e71fabaa82fba4a
>>
>> I hope this helps, and please let us know if you want to know more.
>>
>> Cheers,
>> Aljoscha
>>
>> On Tue, 19 Apr 2016 at 02:11 Yifei Li <lee891031@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am new to Flink and I've read some documentation and think Flink may
>>> fit my scenario.
>>>
>>> Here is my scenario:
>>>
>>> 1. Assume I have 3 streams: S1(id, name, email, action, date), S2(id,
>>> name, email, level, date), S3(id, name, position, date).
>>>
>>> *2. S2 always delays(hours to days, not determined..) *
>>>
>>> 3. Based on the event time, I want to join S1, S2 and S3 every 5
>>> minutes. The join is like a SQL join:
>>>     select S1.name, S3.position from S1, S2, S3 where S1.id = S2.id and
>>> S1.id = S3.id and S1.action = 'download' and S2.level = 5
>>>
>>>
>>>
>>> Can I use Flink for my scenario? Is yes, can anyone point me to some
>>> working examples(I found some examples but they are outdated), or tell me
>>> some workaround to solve this problem? If no, can anyone tell me the
>>> reasons?
>>>
>>> Thanks,
>>>
>>> Yifei
>>>
>>>
>>>
>>>
>>>
>>>
>

Mime
View raw message