hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5603) Ability to disable FileInputFormat listLocatedStatus optimization to save client memory
Date Wed, 30 Oct 2013 19:55:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809537#comment-13809537
] 

Jason Lowe commented on MAPREDUCE-5603:
---------------------------------------

Recently we ran across a jobclient that failed with an OOM error once we updated the cluster
to 0.23.10.  The OOM was triggered by the FileInputFormat listLocatedStatus optimization from
MAPREDUCE-1981, as the client now caches the BlockLocations of all files along with the FileStatus
objects it was caching before.  Normally the user can bump the heap size of the client to
work around this issue.  However if a job has an input with a particularly large number of
BlockLocations, as this job did, it would be nice if the user had the option to disable the
optimization to reduce the required memory necessary for input split calculations.

> Ability to disable FileInputFormat listLocatedStatus optimization to save client memory
> ---------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5603
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5603
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: client, mrv2
>    Affects Versions: 0.23.10, 2.2.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Minor
>
> It would be nice if users had the option to disable the listLocatedStatus optimization
in FileInputFormat to save client memory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message