flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Blocking KeyedCoProcessFunction.processElement1
Date Wed, 29 Jan 2020 13:02:26 GMT
Yes, using AsyncIO could help in the case of blocking operations.

Cheers,
Till

On Tue, Jan 28, 2020 at 10:45 AM Taher Koitawala <taherk77@gmail.com> wrote:

> Would AsyncIO operator not be an option for you to connect to RDBMS?
>
> On Tue, Jan 28, 2020, 12:45 PM Alexey Trenikhun <yender@msn.com> wrote:
>
>> Thank you Yun Tang.
>> My implementation potentially could block for significant amount of time,
>> because I wanted to do RDBMS maintenance (create partitions for new data,
>> purge old data etc) in-line with writing stream data to a database
>>
>> ------------------------------
>> *From:* Yun Tang <myasuka@live.com>
>> *Sent:* Sunday, January 26, 2020 8:42:37 AM
>> *To:* Alexey Trenikhun <yender@msn.com>; user@flink.apache.org <
>> user@flink.apache.org>
>> *Subject:* Re: Blocking KeyedCoProcessFunction.processElement1
>>
>> Hi Alexey
>>
>> Actually, I don't understand why you thing
>> KeyedCoProcessFunction#processElement1 would block for significant amount
>> of time, it just process record from the elements in the first input stream
>> which is necessary. If you really find it would block for a long time, I
>> think that's because your processing logic has some problem to stuck. On
>> the other hand, since processing checkpoint and records hold the same lock,
>> we cannot process checkpoint when the record processing logic did not
>> release the lock.
>>
>> Best
>> Yun Tang
>> ------------------------------
>> *From:* Alexey Trenikhun <yender@msn.com>
>> *Sent:* Thursday, January 23, 2020 13:04
>> *To:* user@flink.apache.org <user@flink.apache.org>
>> *Subject:* Blocking KeyedCoProcessFunction.processElement1
>>
>>
>> Hello,
>> If KeyedCoProcessFunction.processElement1 blocks for significant amount
>> of time, will it prevent checkpoint ?
>>
>> Thanks,
>> Alexey
>>
>

Mime
View raw message