flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Flink JDBCOutputFormat - Flush last batch enhancement
Date Mon, 26 Sep 2016 09:48:29 GMT
Hi!

I am not sure I understand what you want to do, but here are some comments:

  - There is no "batching" in Flink's streaming API, not sure what you are
referring to in with the "last batch"
  - JDBC connections are not closed between windows, they remain open as
long as the operator is open.

Thanks,
Stephan


On Mon, Sep 26, 2016 at 9:29 AM, Swapnil Chougule <the.swapnil09@gmail.com>
wrote:

> Hi Team,
>
> Can we handle one case in connector JDBCOutputFormat to update last batch
> (might be batch count is less than batch interval) without closing jdbc
> connection?
>
> During use case of my streaming project, I am updating jdbc sink (mysql db)
> after every window.
>
> Case : Say I have 450 queries to be updated in mysql with batch interval
> 100.
> 400 queries are executed in 4 batches (4 x 100).
> Last 50 queries go into pending state in batch to be executed & wait for
> next 50 queries from next window.
> If window is of size 5 minutes, then it will take next 4-5 minutes to
> reflect last 50 queries in mysql.
>
> Can we have functionality in JDBCOuputFormat to flush last batch to jdbc
> sink persisting same db connection.?
>
> Thanks,
> Swapnil
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message