beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2601) FileBasedSink produces incorrect shards when writing to multiple destinations
Date Wed, 12 Jul 2017 02:47:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083336#comment-16083336
] 

ASF GitHub Bot commented on BEAM-2601:
--------------------------------------

GitHub user reuvenlax opened a pull request:

    https://github.com/apache/beam/pull/3546

    [BEAM-2601] Fix broken per-destination finalization.

    We now finalize each destination separately. Since temporary-file cleanup happens (for
the non-windowed case) by deleting the entire temp directory, we delay cleanup until all destinations
have been finalized.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/reuvenlax/incubator-beam fix_dynamic_destination_sharding

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3546.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3546
    
----
commit 7254dcf312251a462c70b24c8ff8861c2f009b2b
Author: Reuven Lax <relax@google.com>
Date:   2017-07-12T02:19:42Z

    Fix per-destination finalization.

----


> FileBasedSink produces incorrect shards when writing to multiple destinations
> -----------------------------------------------------------------------------
>
>                 Key: BEAM-2601
>                 URL: https://issues.apache.org/jira/browse/BEAM-2601
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Reuven Lax
>            Assignee: Davor Bonaci
>             Fix For: 2.2.0
>
>
> FileBasedSink now supports multiple dynamic destinations, however it finalizes all files
in a bundle without paying attention to destination. This means that the shard counts will
be incorrect across these destinations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message