beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chamikara Jayalath (JIRA)" <>
Subject [jira] [Assigned] (BEAM-3039) DatastoreIO.Write fails multiple mutations of same entity
Date Wed, 18 Oct 2017 03:55:01 GMT


Chamikara Jayalath reassigned BEAM-3039:

    Assignee: Reuven Lax  (was: Chamikara Jayalath)

> DatastoreIO.Write fails multiple mutations of same entity
> ---------------------------------------------------------
>                 Key: BEAM-3039
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-extensions
>    Affects Versions: 2.3.0
>            Reporter: Alexander Hoem Rosbach
>            Assignee: Reuven Lax
>            Priority: Minor
> When streaming messages from a source that doesn't guarantee once-only-delivery, but
has at-least-once-delivery, then the DatastoreIO.Write will throw an exception which leads
to Dataflow retrying the same commit multiple times before giving up. This leads to a significant
bottleneck in the pipeline, with the end-result that the data is dropped. This should be handled
> There are a number of ways to fix this. One of them could be to drop any duplicate mutations
within one batch. Non-duplicates should also be handled in some way. Perhaps a use NON-TRANSACTIONAL
commit, or make sure the mutations are commited in different commits.
> {code}
> A non-transactional commit may not
contain multiple mutations affecting the same entity., code=INVALID_ARGUMENT

> {code}

This message was sent by Atlassian JIRA

View raw message