beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stas Levin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1050) PipelineResult.State is not set to FAILED in when a streaming job fails
Date Sun, 27 Nov 2016 12:21:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15699605#comment-15699605
] 

Stas Levin commented on BEAM-1050:
----------------------------------

I see that the code has changed since (the relevant snipped is below), but I believe the argument
above still holds.

{code:java}
if (isStreamingPipeline()) {
      // According to PipelineResult: Provide a value less than 1 ms for an infinite wait
      if (duration.getMillis() < 1L) {
        jssc.awaitTermination();
        state = State.DONE;
      } else {
        jssc.awaitTermination(duration.getMillis());
        // According to PipelineResult: The final state of the pipeline or null on timeout
        if (jssc.getState().equals(StreamingContextState.STOPPED)) {
          state = State.DONE;
        } else {
          return null;
        }
      }
      return state;
    } else {
      // This is no-op, since Spark runner in batch is blocking.
      // It needs to be updated once SparkRunner supports non-blocking execution:
      // https://issues.apache.org/jira/browse/BEAM-595
      return State.DONE;
    }
{code}

> PipelineResult.State is not set to FAILED in when a streaming job fails
> -----------------------------------------------------------------------
>
>                 Key: BEAM-1050
>                 URL: https://issues.apache.org/jira/browse/BEAM-1050
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Stas Levin
>            Assignee: Stas Levin
>            Priority: Minor
>
> In case of failure, {{SteamingContext#awaitTerminationOrTimeout}} and {{SteamingContext#awaitTermination}}
will both throw an exception, and so {{state = State.DONE}} will not be executed in the code
block below. 
> In addition, it would probably make sense to set {{state = State.FAILED}} in cases where
an exception takes place.
> {code:java}
> if (isStreamingPipeline()) {
>       // stop streaming context
>       if (timeout > 0) {
>         jssc.awaitTerminationOrTimeout(timeout);
>       } else {
>         jssc.awaitTermination();
>       }
>       // stop streaming context gracefully, so checkpointing (and other computations)
get to
>       // finish before shutdown.
>       jssc.stop(false, gracefully);
>     }
>     state = State.DONE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message