hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-64) Map-side sort is hampered by io.sort.record.percent
Date Sat, 17 Oct 2009 00:51:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766802#action_12766802

Chris Douglas commented on MAPREDUCE-64:

Good ideas; thanks, Todd.

bq. In the configuration parameter, there's the text "Note that this does not imply any chunking
of data to the spill." - I'm not sure what this is trying to say [...]

I was trying to say that, for example, a spill percentage of .3 doesn't mean that collection
will attempt to make its spills at .3 * io.sort.mb. If the spill thread isn't done when the
collection passes .6, it will happily continue collect until the buffer is completely full
or the spill completes. I'll try to clarify the comment.

bq. Consider writing TestMapCollection as a true unit test rather than submitting jobs to
the local cluster [...]
bq. If you do want to submit jobs to the local runner, set Job.COMPLETION_POLL_INTERVAL_KEY

This should be set to 100 in all the jobs (TestMapCollection::runtest(String, Job)), but the
delays are still there... ah, but I'm setting it after the Job is created... OK, I see. I'll
correct that. I agree that TestMapCollection would be better written with mock objects. Once
MAPREDUCE-1050 is resolved, I'll look into it. Perhaps as a separate issue?

bq. suggested test: would be good to have a RecordFactory that generates records of randomized
length to truly stress test the system. If we can figure out a test that generates random
lengths but has output that's still verifiably correct (in terms of contents and not just
total record count) that would be ideal. [...]

This sounds close to TestMiniMRDFSSort, but a random case for the collection unit test makes

bq. For the purpose of continued reviewing (and future documentation) I'd love to see some
kind of diagram of the various (I count 9!) pointers into the data buffer. [...]

There's some related ASCII art in HADOOP-2919, but I'll create some diagrams for this issue
and attach them with a more thorough explanation.

Thanks again for the notes.

> Map-side sort is hampered by io.sort.record.percent
> ---------------------------------------------------
>                 Key: MAPREDUCE-64
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-64
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>            Assignee: Chris Douglas
>         Attachments: M64-0.patch, M64-1.patch, M64-2.patch
> Currently io.sort.record.percent is a fairly obscure, per-job configurable, expert-level
parameter which controls how much accounting space is available for records in the map-side
sort buffer (io.sort.mb). Typically values for io.sort.mb (100) and io.sort.record.percent
(0.05) imply that we can store ~350,000 records in the buffer before necessitating a sort/combine/spill.
> However for many applications which deal with small records e.g. the world-famous wordcount
and it's family this implies we can only use 5-10% of io.sort.mb i.e. (5-10M) before we spill
inspite of having _much_ more memory available in the sort-buffer. The word-count for e.g.
results in ~12 spills (given hdfs block size of 64M). The presence of a combiner exacerbates
the problem by piling serialization/deserialization of records too...
> Sure, jobs can configure io.sort.record.percent, but it's tedious and obscure; we really
can do better by getting the framework to automagically pick it by using all available memory
(upto io.sort.mb) for either the data or accounting.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message