flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Sun" <sunhaib...@163.com>
Subject Re:Re: How to handle JDBC connections in a topology
Date Wed, 24 Jul 2019 08:24:06 GMT
Hi Stephen,

I don't think it's possible to use the same connection pool for the entire topology, because
the nodes on the topology may run in different JVMs and on different machines.

If you want all operators running in the same JVM to use the same connection pool, I think
you can implement a static class that contains the connection pool, and then the operators
get the  connection from it.


At 2019-07-24 15:20:31, "Stephen Connolly" <stephen.alan.connolly@gmail.com> wrote:

Oh and I'd also need some way to clean up the per-node transient state if the topology stops
running on a specific node.

On Wed, 24 Jul 2019 at 08:18, Stephen Connolly <stephen.alan.connolly@gmail.com> wrote:


So we have a number of nodes in our topology that need to do things like checking a database,

* We need a filter step to drop events on the floor from systems we are no longer interested
* We need a step that outputs on a side-channel if the event is for an object where the parent
is not currently known to the database.

Right now we are grabbing a JDBC connection for each node in the topology that needs to talk
to the database and storing the connection in a transient field (to exclude it from the serialized

What I'd really like to do is have a JDBC connection pool shared across the entire topology
as that way we could have the pool check for stale connections, etc.

Does anyone have any tips for doing this kind of thing?

(My current idea is to maintain a `static final WeakHashMap<ClassLoader,ConnectionPool>`
in the main class... but that feels very much like a hack)

What I'm really looking for is some form of Node Transient State... are there any examples
of this type of think.

Flink 1.8.x


View raw message