beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Peterson (JIRA)" <>
Subject [jira] [Resolved] (BEAM-1826) Allow BigqueryIO to forward errors
Date Wed, 29 Mar 2017 21:38:41 GMT


Kevin Peterson resolved BEAM-1826.
       Resolution: Duplicate
    Fix Version/s: Not applicable

> Allow BigqueryIO to forward errors
> ----------------------------------
>                 Key: BEAM-1826
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Kevin Peterson
>            Assignee: Daniel Halperin
>            Priority: Minor
>             Fix For: Not applicable
> Most sinks are terminal - data ends at the sink. While on occasion the sink may temporarily
fail due to resource unavailability, it will eventually succeed. However, some have strict
requirements on this input format. In these cases, retries will never succeed, and continuous
retrying will eventually lead to pipeline failure.
> The primary use case I have in mind is streaming data to a sink such as BigQuery, where
data of the wrong format could fail on insert.
> It would be useful to be able to set a side output or downstream transform from Bigquery
which can receive failed rows where retry will never fix the issue, and allow them to be persisted
to a different output which is more permissive of the output, to prevent data loss.

This message was sent by Atlassian JIRA

View raw message