beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Peterson (JIRA)" <>
Subject [jira] [Created] (BEAM-1826) Allow BigqueryIO to forward errors
Date Wed, 29 Mar 2017 06:39:41 GMT
Kevin Peterson created BEAM-1826:

             Summary: Allow BigqueryIO to forward errors
                 Key: BEAM-1826
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-gcp
            Reporter: Kevin Peterson
            Assignee: Daniel Halperin
            Priority: Minor

Most sinks are terminal - data ends at the sink. While on occasion the sink may temporarily
fail due to resource unavailability, it will eventually succeed. However, some have strict
requirements on this input format. In these cases, retries will never succeed, and continuous
retrying will eventually lead to pipeline failure.

The primary use case I have in mind is streaming data to a sink such as BigQuery, where data
of the wrong format could fail on insert.

It would be useful to be able to set a side output or downstream transform from Bigquery which
can receive failed rows where retry will never fix the issue, and allow them to be persisted
to a different output which is more permissive of the output, to prevent data loss.

This message was sent by Atlassian JIRA

View raw message