hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Kunz (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3841) merge phase runs out of disk space
Date Sat, 13 Sep 2008 18:10:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630809#action_12630809

Christian Kunz commented on HADOOP-3841:

There was another instance of a reducer on a node with more than 280GB free space distributed
over 4 disks and map input of 210 GB that could not find a disk with enough free space when
it tried to create a merge file larger than 30 GB, because it included intermediate merged
files into the new merge attempt although there were enough small non-merged files that could
have been merged to bring the total number of input files below the fan factor.

> merge phase runs out of disk space
> ----------------------------------
>                 Key: HADOOP-3841
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3841
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.2
>            Reporter: Christian Kunz
> We observe that reduce tasks run out of disk space during merging (after fetching all
map output) although there would be enough space if the framework did not try to generate
too large merge files.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message