beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-3484) HadoopInputFormatIO reads big datasets invalid
Date Thu, 19 Apr 2018 15:37:00 GMT


ASF GitHub Bot logged work on BEAM-3484:

                Author: ASF GitHub Bot
            Created on: 19/Apr/18 15:36
            Start Date: 19/Apr/18 15:36
    Worklog Time Spent: 10m 
      Work Description: aromanenko-dev commented on issue #5166: [BEAM-3484] Fix split issue
in HadoopInputFormatIOIT
   Run Java PreCommit

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 92632)
    Time Spent: 1h  (was: 50m)

> HadoopInputFormatIO reads big datasets invalid
> ----------------------------------------------
>                 Key: BEAM-3484
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-hadoop
>    Affects Versions: 2.3.0, 2.4.0
>            Reporter: Łukasz Gajowy
>            Assignee: Alexey Romanenko
>            Priority: Minor
>             Fix For: 2.5.0
>         Attachments: result_sorted1000000, result_sorted600000
>          Time Spent: 1h
>  Remaining Estimate: 0h
> For big datasets HadoopInputFormat sometimes skips/duplicates elements from database
in resulting PCollection. This gives incorrect read result.
> Occurred to me while developing HadoopInputFormatIOIT and running it on dataflow. For
datasets smaller or equal to 600 000 database rows I wasn't able to reproduce the issue. Bug
appeared only for bigger sets, eg. 700 000, 1 000 000. 
> Attachments:
>   - text file with sorted result saved using TextIO.write().to().withoutSharding().
If you look carefully you'll notice duplicates or missing values that should not happen
>  - same text file for 600 000 records not having any duplicates and missing elements
>  - link to a PR with HadoopInputFormatIO integration test that allows to reproduce this
issue. At the moment of writing, this code is not merged yet.

This message was sent by Atlassian JIRA

View raw message