flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JingsongLee" <lzljs3620...@aliyun.com>
Subject Re: Streaming write to Hive
Date Fri, 06 Sep 2019 02:52:49 GMT
Hi luoqi:

With partition support[1], I want to introduce a FileFormatSink to
 cover streaming exactly-once and partition-related logic for flink
 file connectors and hive connector. You can take a look.

[1] https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing

Jingsong Lee

From:Bowen Li <bowenli86@gmail.com>
Send Time:2019年9月6日(星期五) 05:21
To:Qi Luo <luoqi.bd@gmail.com>
Cc:user <user@flink.apache.org>; snake.fly318 <snake.fly318@gmail.com>; lichang.bd
Subject:Re: Streaming write to Hive


I'm not sure if there's one yet. Feel free to create one if not.
On Wed, Sep 4, 2019 at 11:28 PM Qi Luo <luoqi.bd@gmail.com> wrote:

Hi Bowen,

Thank you for the information! Streaming write to Hive is a very common use case for our users.
Is there any open issue for this to which we can try contributing?

+Yufei and Chang who are also interested in this.

On Thu, Sep 5, 2019 at 12:16 PM Bowen Li <bowenli86@gmail.com> wrote:
Hi Qi,

With 1.9 out of shelf, I'm afraid not. You can make HiveTableSink implements AppendStreamTableSink
(an empty interface for now) so it can be picked up in streaming job. Also, streaming requires
checkpointing, and Hive sink doesn't do that yet. There might be other tweaks you need to

It's on our list for 1.10, not high priority though.

On Wed, Sep 4, 2019 at 2:23 AM Qi Luo <luoqi.bd@gmail.com> wrote:
Hi guys,

In Flink 1.9 HiveTableSink is added to support writing to Hive, but it only supports batch
mode. StreamingFileSink can write to HDFS in streaming mode, but it has no Hive related functionality
(e.g. adding Hive partition).

Is there any easy way we can streaming write to Hive (with exactly-once guarantee)?


View raw message