flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JingsongLee" <lzljs3620...@aliyun.com>
Subject Re: Streaming write to Hive
Date Fri, 06 Sep 2019 02:52:49 GMT
Hi luoqi:

With partition support[1], I want to introduce a FileFormatSink to
 cover streaming exactly-once and partition-related logic for flink
 file connectors and hive connector. You can take a look.

[1] https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing

Best,
Jingsong Lee


------------------------------------------------------------------
From:Bowen Li <bowenli86@gmail.com>
Send Time:2019年9月6日(星期五) 05:21
To:Qi Luo <luoqi.bd@gmail.com>
Cc:user <user@flink.apache.org>; snake.fly318 <snake.fly318@gmail.com>; lichang.bd
<lichang.bd@gmail.com>
Subject:Re: Streaming write to Hive

Hi, 

I'm not sure if there's one yet. Feel free to create one if not.
On Wed, Sep 4, 2019 at 11:28 PM Qi Luo <luoqi.bd@gmail.com> wrote:

Hi Bowen,

Thank you for the information! Streaming write to Hive is a very common use case for our users.
Is there any open issue for this to which we can try contributing?

+Yufei and Chang who are also interested in this.

Thanks,
Qi
On Thu, Sep 5, 2019 at 12:16 PM Bowen Li <bowenli86@gmail.com> wrote:
Hi Qi,

With 1.9 out of shelf, I'm afraid not. You can make HiveTableSink implements AppendStreamTableSink
(an empty interface for now) so it can be picked up in streaming job. Also, streaming requires
checkpointing, and Hive sink doesn't do that yet. There might be other tweaks you need to
make.

It's on our list for 1.10, not high priority though.

Bowen
On Wed, Sep 4, 2019 at 2:23 AM Qi Luo <luoqi.bd@gmail.com> wrote:
Hi guys,

In Flink 1.9 HiveTableSink is added to support writing to Hive, but it only supports batch
mode. StreamingFileSink can write to HDFS in streaming mode, but it has no Hive related functionality
(e.g. adding Hive partition).

Is there any easy way we can streaming write to Hive (with exactly-once guarantee)?

Thanks,
Qi

Mime
View raw message