hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1363) Spill size underestimated when using certain combiners, causing map tasks to fail
Date Thu, 07 Jan 2010 23:06:54 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797827#action_12797827
] 

Chris Douglas commented on MAPREDUCE-1363:
------------------------------------------

bq. This could be avoided by either improving the estimation of the size of the spill (increasing
it by a constant amount or by constant percentage), or LocalDirAllocator could take into consideration
a configuration parameter specifying how much extra unused space should be on the path returned
by getLocalPathForWrite

Yet another esoteric, static configuration parameter would be an unfortunate approach and
a fixed percentage is patently wrong (wordcount-like applications will overestimate, not underestimate
space requirements). A decent fix could estimate and adapt requests for space based on the
expansion/contraction observed in prior spills. Most combiners' ratio won't be a simple scalar,
but a function of key distribution during both the collection and merge phases; this ratio
may not be stable between those phases, either. That said, it's not a bad place to start.

One approach could initialize a float using a configurable ratio (defaulting to 1.0) adjusted
at each spill. This could naively update a global average for each spill, use an exponential
moving average to avoid radical shifts due to abnormally large records, odd skews, etc. Whether
the ratio is tracked for each invocation of the combiner and not over the entire spill doesn't
seem too important if the ratio is stable, which this implicitly assumes. Tracking per-partition
expansion/contraction is probably more correct for some applications, but of dubious value
generally.

bq. In case there is no space left on a device designated for writing intermediate data on,
the spill could be retried on a different device (without the failure of the map task).

This is slightly tricky, as one would want to copy the data already spilled to the end of
the prior partition and re-run the combiner on the partition that caused the error (since
partially serialized data causes all sorts of problems) before continuing the spill.

> Spill size underestimated when using certain combiners, causing map tasks to fail
> ---------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1363
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1363
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>    Affects Versions: 0.20.1
>            Reporter: Miklos Erdelyi
>
> Spill size could get underestimated when using certain combiners, causing map tasks to
fail when disk space is really low in some mapred.local.dir.
> When doing sortAndSpill(), MapOutputBuffer gets an output path through LocalDirAllocator
which checks if the estimated size of the spill is available on any paths specified for intermediate
data storage. In case a combiner is specified which emits key-value pairs having serialized
size larger than the input key-value pairs' size, the size of the spill file is underestimated.
If LocalDirAllocator selects a path for intermediate data storage which does not have enough
space to hold the spilled records, an IOException is thrown and the map task fails.
> This could be avoided by either improving the estimation of the size of the spill (increasing
it by a constant amount or by constant percentage), or LocalDirAllocator could take into consideration
a configuration parameter specifying how much extra unused space should be on the path returned
by getLocalPathForWrite (similarly to dfs.datanode.du.reserved). In case there is no space
left on a device designated for writing intermediate data on, the spill could be retried on
a different device (without the failure of the map task).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message