crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hector Izquierdo Seliva (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-463) Copying the Configuration object in every CrunchInputSplit causes OOM errors for jobs with lots of splits
Date Thu, 14 Aug 2014 14:28:11 GMT
Hector Izquierdo Seliva created CRUNCH-463:
----------------------------------------------

             Summary: Copying the Configuration object in every CrunchInputSplit causes OOM
errors for jobs with lots of splits
                 Key: CRUNCH-463
                 URL: https://issues.apache.org/jira/browse/CRUNCH-463
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.10.0
            Reporter: Hector Izquierdo Seliva
            Assignee: Josh Wills


Trying to run a job with 11k input files and that yields about 25k splits results in OOM errors
due to too many copies of the Configuration object being created when the CrunchInputSplit
is initialised. I know that that's the result of CRUNCH-313, but perhaps a better way to deal
with that problem should be found.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message