Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
From: Kostas Kloudas <k.kloudas@data-artisans.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.1 \(3251\))
Subject: [DISCUSS] Adding a dispose() method in the RichFunction.
Message-Id: <30754B18-717E-4AC4-B215-9E39E112D84F@data-artisans.com>
Date: Thu, 10 Nov 2016 11:37:36 +0100
To: dev@flink.apache.org
archived-at: Thu, 10 Nov 2016 10:38:16 -0000

Hello,

I would like to propose the addition of a dispose() method, in addition =
to the=20
already existing close(), in the RichFunction interface. This will align =
the lifecycle=20
of a RichFunction, with that of an Operator. After this, the code paths =
followed=20
when finishing successfully and when cancelling, will be totally =
distinct.

Semantically, close() will be responsible for guaranteeing semantic =
correctness=20
of the job when the job terminates successfully, while dispose() will be =
responsible=20
for taking care of system clean up both when terminating gracefully and =
when=20
cancelling, e.g. freeing resources like db connections.=20

Currently, most functions use close() with the semantics of dispose() as =
the only=20
thing they do is freeing up resources. A nice example where this leads =
to confusion=20
is the case of the BucketingSink/RollingSink where at close(), data that =
is not
committed is marked as "pending" (semantic correctness of the job). In =
this case,=20
and given that there is no distinction between close() and dispose(), =
this method is=20
called by the AbstractUdfStreamOperator both when successfully finishing =
a job and=20
when something went wrong during execution. This is essentially a =
compromise,=20
as the close() should mark this data as "committed" when successfully =
terminating,=20
but it cannot as it can also be called when an exception was thrown.

Let me know what you think,
Kostas=20=