flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Papadopoulos, Konstantinos" <Konstantinos.Papadopou...@IRIworldwide.com>
Subject RE: Flink JDBC: Disable auto-commit mode
Date Mon, 15 Apr 2019 06:53:23 GMT
Hi Rong,

We have already tried to set the fetch size with no success. According to PG documentation
we have to set both configuration parameters (i.e., auto-commit to false and limit fetch)
to achieve our purpose.

Thanks,
Konstantinos

From: Rong Rong <walterddr@gmail.com>
Sent: Παρασκευή, 12 Απριλίου 2019 6:50 μμ
To: Papadopoulos, Konstantinos <Konstantinos.Papadopoulos@IRIworldwide.com>
Cc: user <user@flink.apache.org>
Subject: Re: Flink JDBC: Disable auto-commit mode

Hi Konstantinos,

Seems like setting for auto commit is not directly possible in the current JDBCInputFormatBuilder.
However there's a way to specify the fetch size [1] for your DB round-trip, doesn't that resolve
your issue?

Similarly in JDBCOutputFormat, a batching mode was also used to stash upload rows before flushing
to DB.

--
Rong

[1] https://docs.oracle.com/cd/E18283_01/java.112/e16548/resltset.htm#insertedID4<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.oracle.com%2Fcd%2FE18283_01%2Fjava.112%2Fe16548%2Fresltset.htm%23insertedID4&data=02%7C01%7C%7C44d9fa279b674725483008d6bf5e7b3e%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C1%7C636906809861700178&sdata=XE%2FjZhh%2F8dcBe679%2FccdGq0rutlAiXHBHTj45eoXM70%3D&reserved=0>

On Fri, Apr 12, 2019 at 6:23 AM Papadopoulos, Konstantinos <Konstantinos.Papadopoulos@iriworldwide.com<mailto:Konstantinos.Papadopoulos@iriworldwide.com>>
wrote:
Hi all,

We are facing an issue when trying to integrate PostgreSQL with Flink JDBC. When you establish
a connection to the PostgreSQL database, it is in auto-commit mode. It means that each SQL
statement is treated as a transaction and is automatically committed, but this functionality
results in unexpected behavior (e.g., out-of-memory errors) when executed for large result
sets. In order to bypass such issues, we must disable the auto-commit mode. To do this, in
a simple Java application, we call the setAutoCommit() method of the Connection object.

So, my question is: How can we achieve this by using JDBCInputFormat of Flink?

Thanks in advance,

Konstantinos
Mime
View raw message