beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reuven Lax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1438) The default behavior for the Write transform doesn't work well with the Dataflow streaming runner
Date Wed, 21 Jun 2017 01:49:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056813#comment-16056813
] 

Reuven Lax commented on BEAM-1438:
----------------------------------

I believe so




> The default behavior for the Write transform doesn't work well with the Dataflow streaming
runner
> -------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-1438
>                 URL: https://issues.apache.org/jira/browse/BEAM-1438
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>            Reporter: Reuven Lax
>            Assignee: Reuven Lax
>
> If a Write specifies 0 output shards, that implies the runner should pick an appropriate
sharding. The default behavior is to write one shard per input bundle. This works well with
the Dataflow batch runner, but not with the streaming runner which produces large numbers
of small bundles.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message