flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Connolly <stephen.alan.conno...@gmail.com>
Subject Re: How to handle JDBC connections in a topology
Date Wed, 24 Jul 2019 07:20:31 GMT
Oh and I'd also need some way to clean up the per-node transient state if
the topology stops running on a specific node.

On Wed, 24 Jul 2019 at 08:18, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:

> Hi,
>
> So we have a number of nodes in our topology that need to do things like
> checking a database, e.g.
>
> * We need a filter step to drop events on the floor from systems we are no
> longer interested in
> * We need a step that outputs on a side-channel if the event is for an
> object where the parent is not currently known to the database.
>
> Right now we are grabbing a JDBC connection for each node in the topology
> that needs to talk to the database and storing the connection in a
> transient field (to exclude it from the serialized state)
>
> What I'd really like to do is have a JDBC connection pool shared across
> the entire topology as that way we could have the pool check for stale
> connections, etc.
>
> Does anyone have any tips for doing this kind of thing?
>
> (My current idea is to maintain a `static final
> WeakHashMap<ClassLoader,ConnectionPool>` in the main class... but that
> feels very much like a hack)
>
> What I'm really looking for is some form of Node Transient State... are
> there any examples of this type of think.
>
> Flink 1.8.x
>
> Thanks,
>
> -Stephen
>

Mime
View raw message