beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chamikara Jayalath <chamik...@google.com>
Subject Re: Cleanup resources on pipeline cancelation
Date Wed, 01 Aug 2018 14:09:44 GMT
Hi Andrew,

Beam currently does not have a generalized cleanup story so answer usually
has been ad-hoc. For bounded source we can (1) cleanup any resources
created for splitting after splitting (2) cleanup resources created for a
given reader when the reader exists (last advaince() call).

I'm not sure what the proper solution for UnboundedSources is and it might
not even make sense to to add cleanup logic to an unbounded source that is
never expected to end. We might need something more generic (for example, a
mechanism to collect temporary resources and delete such resources at
pipeline termination).

Thanks,
Cham


On Tue, Jul 31, 2018 at 10:04 PM Romain Manni-Bucau <rmannibucau@gmail.com>
wrote:

> Hi Andrew,
>
> IIRC sources should clean up their resources per method since they dont
> have a better lifecycle. Readers can create anything longer and release it
> at close time.
>
>
> Le mer. 1 août 2018 00:31, Andrew Pilloud <apilloud@google.com> a écrit :
>
>> Some of our IOs create external resources that need to be cleaned up when
>> a pipeline is terminated. It looks like the
>> org.apache.beam.sdk.io.UnboundedSource interface is called on creation, but
>> there is no call for cleanup. For example, PubsubIO creates a Pubsub
>> subcription in createReader()/split() and it should be deleted at shutdown.
>> Does anyone have ideas on how I might make this happen?
>>
>> (I filed https://issues.apache.org/jira/browse/BEAM-5051 tracking the
>> PubSub specific issue.)
>>
>> Andrew
>>
>

Mime
View raw message