beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-10706) DynamoDBIO fail to write to the same key in short consecution
Date Sat, 05 Dec 2020 06:45:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-10706?focusedWorklogId=520440&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520440
]

ASF GitHub Bot logged work on BEAM-10706:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Dec/20 06:44
            Start Date: 05/Dec/20 06:44
    Worklog Time Spent: 10m 
      Work Description: dennisylyung commented on pull request #12583:
URL: https://github.com/apache/beam/pull/12583#issuecomment-739135639


   > My excuses @dennisylyung I took a quick look today and it looks mostly ok (I have
some minor things that I can fix) the only issue I have is that I just realized that the tests
are disabled for DynamoDB how did you get them to run? I removed the Ignore annotation and
enabled the specific extra test on this PR but I get a failure.
   
   I tested also by commenting out the ignore annotation.
   The test itself passes, but `destroy` fails due to this bug https://github.com/localstack/localstack/issues/2702.
The error is universal to all tests in DynamoDBIOTest, not just the new one.
   
   I upgraded Localstack to the fixed version and it now passes. (It always passes in aws2)
@iemejia 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 520440)
    Time Spent: 3h 20m  (was: 3h 10m)

> DynamoDBIO fail to write to the same key in short consecution
> -------------------------------------------------------------
>
>                 Key: BEAM-10706
>                 URL: https://issues.apache.org/jira/browse/BEAM-10706
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-aws
>    Affects Versions: 2.23.0
>            Reporter: Dennis Yung
>            Assignee: Dennis Yung
>            Priority: P2
>             Fix For: 2.27.0
>
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Internally, DynamoDBIO.Write uses the batchWriteItem method from the AWS SDK to sink
items. However, there is a limitation in the AWS SDK that a call to batchWriteItem cannot
contain duplicate keys.
> Currently DynamoDBIO.Write performs no key deduplication before flushing a batch, which
could cause ValidationException: Provided list of item keys contains duplicates, if consecutive
updates to a single key is within the batch size (currently hardcoded to be 25). 
> To fix this bug, the batch of write requests need to be deduplicated before being sent
to batchRequest.addRequestItemsEntry



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message