beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pei He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources
Date Tue, 04 Oct 2016 19:33:21 GMT

    [ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546411#comment-15546411
] 

Pei He commented on BEAM-702:
-----------------------------

1. If there are failures in one bundle, we can easily call teardown()s of DoFns associated
with this bundle. (I think we had plan to do it, I am not sure it has been done or not.)
The tricky part is that if there are failures in other PTransforms or in the environment while
we are processing the bundles. Before runners fail the whole job, we want runners be able
to call teardown() for all running ParDos and their bundles in processing. We don't have a
plan for this part yet.

2. We should document and clarify whether finishBundle() will be called or not if the bundle
fails.

> Simple pattern for per-bundle and per-DoFn Closeable resources
> --------------------------------------------------------------
>
>                 Key: BEAM-702
>                 URL: https://issues.apache.org/jira/browse/BEAM-702
>             Project: Beam
>          Issue Type: Improvement
>            Reporter: Eugene Kirpichov
>
> Dealing with Closeable resources inside a processElement call is easy: simply use try-with-resources.
> However, bundle- or DoFn-scoped resources, such as long-lived database connections, are
less convenient to deal with: you have to open them in startBundle and conditionally close
in finishBundle (likewise setup/teardown), taking special care if there's multiple resources
to close all of them.
> Perhaps we should provide something like Guava's Closer to DoFn's https://github.com/google/guava/wiki/ClosingResourcesExplained.
Ideally, the user would need to only write a startBundle() or setup() method, but not write
finishBundle() or teardown() - resources would be closed automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message