hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2117) mapred should provide an optional upper-bound for size of map outputs
Date Sun, 28 Oct 2007 19:44:50 GMT
mapred should provide an optional upper-bound for size of map outputs
---------------------------------------------------------------------

                 Key: HADOOP-2117
                 URL: https://issues.apache.org/jira/browse/HADOOP-2117
             Project: Hadoop
          Issue Type: Improvement
          Components: mapred
    Affects Versions: 0.14.3
         Environment: All
            Reporter: Milind Bhandarkar
             Fix For: 0.16.0


AFAIK, currently, there is no explicit limit on the amount of disk space available for map
tasks to emit map-outputs. They can use whatever space is available (This corresponds to total
disk space minus disk space currently used by data-nodes to store dfs blocks.) Thus, some
jobs that run fine when DFS is less full, stop working when dfs becomes more full.

If mapred had an (optional) upper-limit on the amount of local disk space used for map outputs,
these jobs will run more predictably.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message