beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3194) Support annotating that a DoFn requires stable / deterministic input for replay/retry
Date Tue, 28 Nov 2017 20:33:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269417#comment-16269417
] 

ASF GitHub Bot commented on BEAM-3194:
--------------------------------------

kennknowles commented on a change in pull request #4135: [BEAM-3194] Add @RequiresStableInput
annotation
URL: https://github.com/apache/beam/pull/4135#discussion_r153614701
 
 

 ##########
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
 ##########
 @@ -568,6 +568,29 @@ public Duration getAllowedTimestampSkew() {
   public @interface ProcessElement {}
 
   /**
+   * <b><i>Experimental - no backwards compatibility guarantees. The exact name
or usage of this
+   * feature may change.</i></b>
+   *
+   * <p>Annotation that may be added to a {@link ProcessElement} or {@link OnTimer}
method to
+   * indicate that the observable contents of the input {@link PCollection} or mutable state
must be
+   * stable upon retries.
+   *
+   * <p>This is important for exactly-once semantics when writing to a storage medium
outside of
+   * your pipeline. The general pattern is to establish an idempotent protocol for writing
the data,
+   * and then require stable input. Combined, these allow the write to be freely retried
until
+   * success.
+   *
+   * <p>An example of an unstable input would be anything computed using nondeterministic
logic. In
+   * Beam, any user-defined function is permitted to be nondeterministic, and any {@link
+   * PCollection} is permitted to be recomputed in any manner.
+   */
+  @Documented
+  @Experimental
+  @Retention(RetentionPolicy.RUNTIME)
+  @Target(ElementType.METHOD)
+  public @interface RequiresStableInput {}
 
 Review comment:
   I think you could go either way, but I think this one is a bit more flexible - no need
to add the flag to various places you might add it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Support annotating that a DoFn requires stable / deterministic input for replay/retry
> -------------------------------------------------------------------------------------
>
>                 Key: BEAM-3194
>                 URL: https://issues.apache.org/jira/browse/BEAM-3194
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Kenneth Knowles
>            Assignee: Kenneth Knowles
>
> See the thread: https://lists.apache.org/thread.html/5fd81ce371aeaf642665348f8e6940e308e04275dd7072f380f9f945@%3Cdev.beam.apache.org%3E
> We need this in order to have truly cross-runner end-to-end exactly once via replay +
idempotence.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message