beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors
Date Thu, 24 Aug 2017 21:01:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Davor Bonaci reassigned BEAM-2302:
----------------------------------

    Assignee: Reuven Lax  (was: Davor Bonaci)

> WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors
> -----------------------------------------------------------------------------------------
>
>                 Key: BEAM-2302
>                 URL: https://issues.apache.org/jira/browse/BEAM-2302
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Reuven Lax
>            Assignee: Reuven Lax
>
> This is because the WriteWindowedBundles transform will create many file writers, and
the sheer number of file buffers (which defaults to 64mb per writer) uses up all memory. The
fix is the same as was done in BigQueryIO - if too many writers are opened, spill into a shuffle,
and write the files after the shuffle



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message