hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-64) Map-side sort is hampered by io.sort.record.percent
Date Sun, 18 Oct 2009 04:04:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767009#action_12767009
] 

Todd Lipcon commented on MAPREDUCE-64:
--------------------------------------

Thanks for those changes.

Regarding diagrams, I had an idea - one simple way might be to simply add TRACE level log
messages at every collect() call with the current values of every index plus the spill number
in simple TSV format. Feeding those into gnuplot or R to graph them should produce something
reasonably informative without much manual diagramming. If you want, I can try to do this
and post a patch - just let me know. My inspiration here was seekwatcher: http://oss.oracle.com/~mason/seekwatcher/ext3_vs_btrfs_vs_xfs.png

Also, I remember when we were first looking at this last August, I ran a simple test where
I was running a sort of 10 byte records, and it turned out that the "optimal" io.sort.record.percent
caused my job to be significantly slower. It was the case then that a small number of large
spills actually ran slower than a large number of small spills. Did we ever determine what
that issue was? I think we should try to understand why the theory isn't agreeing with observations
here.

> Map-side sort is hampered by io.sort.record.percent
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-64
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-64
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>            Assignee: Chris Douglas
>         Attachments: M64-0.patch, M64-1.patch, M64-2.patch, M64-3.patch
>
>
> Currently io.sort.record.percent is a fairly obscure, per-job configurable, expert-level
parameter which controls how much accounting space is available for records in the map-side
sort buffer (io.sort.mb). Typically values for io.sort.mb (100) and io.sort.record.percent
(0.05) imply that we can store ~350,000 records in the buffer before necessitating a sort/combine/spill.
> However for many applications which deal with small records e.g. the world-famous wordcount
and it's family this implies we can only use 5-10% of io.sort.mb i.e. (5-10M) before we spill
inspite of having _much_ more memory available in the sort-buffer. The word-count for e.g.
results in ~12 spills (given hdfs block size of 64M). The presence of a combiner exacerbates
the problem by piling serialization/deserialization of records too...
> Sure, jobs can configure io.sort.record.percent, but it's tedious and obscure; we really
can do better by getting the framework to automagically pick it by using all available memory
(upto io.sort.mb) for either the data or accounting.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message