crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CRUNCH-463) Copying the Configuration object in every CrunchInputSplit causes OOM errors for jobs with lots of splits
Date Tue, 19 Aug 2014 17:51:18 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Wills resolved CRUNCH-463.
-------------------------------

       Resolution: Fixed
    Fix Version/s: 0.11.0
                   0.8.4

Fixed on master and 0.8. Thanks for the heads up, [~hekoru]!

> Copying the Configuration object in every CrunchInputSplit causes OOM errors for jobs
with lots of splits
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-463
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-463
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.10.0
>            Reporter: Hector Izquierdo Seliva
>            Assignee: Josh Wills
>             Fix For: 0.8.4, 0.11.0
>
>         Attachments: CRUNCH-463.patch
>
>
> Trying to run a job with 11k input files and that yields about 25k splits results in
OOM errors due to too many copies of the Configuration object being created when the CrunchInputSplit
is initialised. I know that that's the result of CRUNCH-313, but perhaps a better way to deal
with that problem should be found.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message