flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Swapnil Chougule <the.swapni...@gmail.com>
Subject Flink JDBCOutputFormat - Flush last batch enhancement
Date Mon, 26 Sep 2016 07:29:57 GMT
Hi Team,

Can we handle one case in connector JDBCOutputFormat to update last batch
(might be batch count is less than batch interval) without closing jdbc
connection?

During use case of my streaming project, I am updating jdbc sink (mysql db)
after every window.

Case : Say I have 450 queries to be updated in mysql with batch interval
100.
400 queries are executed in 4 batches (4 x 100).
Last 50 queries go into pending state in batch to be executed & wait for
next 50 queries from next window.
If window is of size 5 minutes, then it will take next 4-5 minutes to
reflect last 50 queries in mysql.

Can we have functionality in JDBCOuputFormat to flush last batch to jdbc
sink persisting same db connection.?

Thanks,
Swapnil

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message