beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors
Date Thu, 24 Aug 2017 21:00:06 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140689#comment-16140689
] 

Davor Bonaci commented on BEAM-2302:
------------------------------------

Fixed?

> WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors
> -----------------------------------------------------------------------------------------
>
>                 Key: BEAM-2302
>                 URL: https://issues.apache.org/jira/browse/BEAM-2302
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Reuven Lax
>            Assignee: Davor Bonaci
>
> This is because the WriteWindowedBundles transform will create many file writers, and
the sheer number of file buffers (which defaults to 64mb per writer) uses up all memory. The
fix is the same as was done in BigQueryIO - if too many writers are opened, spill into a shuffle,
and write the files after the shuffle



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message