Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <A243DE8E-1AD8-4871-8E1C-D00215209926@data-artisans.com>
References: <CAAc9rOy1Vs_5u8t7tZvA2S3GOsF2dc8gvo63GtCFeDrHN7yaoA@mail.gmail.com>
 <A243DE8E-1AD8-4871-8E1C-D00215209926@data-artisans.com>
From: Jinhua Luo <luajit.io@gmail.com>
Date: Fri, 8 Dec 2017 21:37:55 +0800
Message-ID: <CAAc9rOxOKPgSZxCVLuZH5nodAOHnAJ4QuEHTJqVgPnCjsuQ8bQ@mail.gmail.com>
Subject: Re: does the flink sink only support bio?
To: Stefan Richter <s.richter@data-artisans.com>
Cc: user@flink.apache.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 08 Dec 2017 13:38:01 -0000

Thank you very much!

I have two new questions:

1) the async operator must emit some value to the async collector
(even it acts as a sink), right?

2) How could I use CheckpointListener with async operator? Could you
give a simple example or doc page?


2017-12-08 18:25 GMT+08:00 Stefan Richter <s.richter@data-artisans.com>:
> Hi,
>
> Flink currently does not offer async sinks out of the box, but there is n=
o fundamental problem against having them and we will probably offer someth=
ing is this direction in the future. In the meantime, you can build somethi=
ng like this by replacing the sink with an async io operator that acts as s=
ink (i.e. does the writes to the db) followed by a discarding sink for comp=
liance with the API.
>
> You need to be a bit careful if your sink needs exactly-once semantics. I=
n this case things should either be idempotent or the db must support rolli=
ng back changes between checkpoints, e.g. via transactions. Commits should =
be triggered for confirmed checkpoints (=E2=80=9EnotifyCheckpointComplete=
=E2=80=9C).
>
> Your assumptions about the blocking behavior of the non-async sinks is co=
rrect.
>
> Best,
> Stefan
>
>> Am 08.12.2017 um 08:11 schrieb Jinhua Luo <luajit.io@gmail.com>:
>>
>> Hi, all.
>>
>> The invoke method of sink seems no way to make async io? e.g. returns Fu=
ture?
>>
>> For example, the redis connector uses jedis lib to execute redis
>> command synchronously:
>>
>> https://github.com/apache/bahir-flink/blob/master/flink-connector-redis/=
src/main/java/org/apache/flink/streaming/connectors/redis/RedisSink.java
>>
>> Then it will block the task thread of flink waiting the network
>> response from redis server per command?! Is it possible for other
>> operators running in the same thread with sink? If so, then it would
>> block them too?
>>
>> I know flink has asyncio api, but it seems not for used by sink impl?
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/a=
syncio.html
>>
>> Thanks.
>