flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Flink JDBC: Disable auto-commit mode
Date Mon, 15 Apr 2019 14:33:55 GMT
Great, thank you!

Am Mo., 15. Apr. 2019 um 16:28 Uhr schrieb Papadopoulos, Konstantinos <
Konstantinos.Papadopoulos@iriworldwide.com>:

> Hi Fabian,
>
>
>
> I opened the following issue to track the improvement proposed:
>
> https://issues.apache.org/jira/browse/FLINK-12198
>
>
>
> Best,
>
> Konstantinos
>
>
>
> *From:* Papadopoulos, Konstantinos
> <Konstantinos.Papadopoulos@IRIworldwide.com>
> *Sent:* Δευτέρα, 15 Απριλίου 2019 12:30 μμ
> *To:* Fabian Hueske <fhueske@gmail.com>
> *Cc:* Rong Rong <walterddr@gmail.com>; user <user@flink.apache.org>
> *Subject:* RE: Flink JDBC: Disable auto-commit mode
>
>
>
> Hi Fabian,
>
>
>
> Glad to hear that you agree for such an improvement. Of course, I can
> handle it.
>
>
>
> Best,
>
> Konstantinos
>
>
>
> *From:* Fabian Hueske <fhueske@gmail.com>
> *Sent:* Δευτέρα, 15 Απριλίου 2019 11:56 πμ
> *To:* Papadopoulos, Konstantinos <
> Konstantinos.Papadopoulos@IRIworldwide.com>
> *Cc:* Rong Rong <walterddr@gmail.com>; user <user@flink.apache.org>
> *Subject:* Re: Flink JDBC: Disable auto-commit mode
>
>
>
> Hi Konstantinos,
>
>
>
> This sounds like a useful extension to me.
>
> Would you like to create a Jira issue and contribute the improvement?
>
>
>
> In the meantime, you can just fork the code of JDBCInputFormat and adjust
> it to your needs.
>
>
>
> Best, Fabian
>
>
>
> Am Mo., 15. Apr. 2019 um 08:53 Uhr schrieb Papadopoulos, Konstantinos <
> Konstantinos.Papadopoulos@iriworldwide.com>:
>
> Hi Rong,
>
>
>
> We have already tried to set the fetch size with no success. According to
> PG documentation we have to set both configuration parameters (i.e.,
> auto-commit to false and limit fetch) to achieve our purpose.
>
>
>
> Thanks,
>
> Konstantinos
>
>
>
> *From:* Rong Rong <walterddr@gmail.com>
> *Sent:* Παρασκευή, 12 Απριλίου 2019 6:50 μμ
> *To:* Papadopoulos, Konstantinos <
> Konstantinos.Papadopoulos@IRIworldwide.com>
> *Cc:* user <user@flink.apache.org>
> *Subject:* Re: Flink JDBC: Disable auto-commit mode
>
>
>
> Hi Konstantinos,
>
>
>
> Seems like setting for auto commit is not directly possible in the current
> JDBCInputFormatBuilder.
>
> However there's a way to specify the fetch size [1] for your DB
> round-trip, doesn't that resolve your issue?
>
>
>
> Similarly in JDBCOutputFormat, a batching mode was also used to stash
> upload rows before flushing to DB.
>
>
>
> --
>
> Rong
>
>
>
> [1]
> https://docs.oracle.com/cd/E18283_01/java.112/e16548/resltset.htm#insertedID4
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.oracle.com%2Fcd%2FE18283_01%2Fjava.112%2Fe16548%2Fresltset.htm%23insertedID4&data=02%7C01%7C%7C2cc5777737b740b23dcd08d6c185070b%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636909174427110104&sdata=vHwPnVT%2BI41Xxkp1Zfl%2BOgTReZ0ILL5RkhDez72jJvM%3D&reserved=0>
>
>
>
> On Fri, Apr 12, 2019 at 6:23 AM Papadopoulos, Konstantinos <
> Konstantinos.Papadopoulos@iriworldwide.com> wrote:
>
> Hi all,
>
> We are facing an issue when trying to integrate PostgreSQL with Flink
> JDBC. When you establish a connection to the PostgreSQL database, it is in
> auto-commit mode. It means that each SQL statement is treated as a
> transaction and is automatically committed, but this functionality results
> in unexpected behavior (e.g., out-of-memory errors) when executed for large
> result sets. In order to bypass such issues, we must disable the
> auto-commit mode. To do this, in a simple Java application, we call the
> setAutoCommit() method of the Connection object.
>
> So, my question is: How can we achieve this by using JDBCInputFormat of
> Flink?
>
> Thanks in advance,
>
> Konstantinos
>
>

Mime
View raw message