split reduce compute phase ioto two threads one for reading and another for computing
-------------------------------------------------------------------------------------
Key: MAPREDUCE-1939
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1939
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.20.2
Reporter: wangxiaowei
Fix For: 0.20.2
it is known that reduce task is made up of three phases: shuffle , sort and reduce. During
reduce phase, a reduce function will read a record from disk or memory first and process it
to write to hdfs finally. To convert this serial progress to parallel progress , I split the
reduce phase into two threads called producer and consumer individually. producer is used
to read record from disk and consumer to process the records read by the first one. I use
two buffer, if producer is writing one buffer consumer will read from another buffer. Theoretically
there will be a overlap between this two phases so we can reduce the whole reduce time.
I wonder why hadoop does not implement it originally? Is there some potential problems for
such ideas ?
I have already implemmented a prototypy. The producer just reads bytes from the disk and leaves
the work of transformation to real key and value objects to consumer. The results is not good
only a improvement of 13% for time. I think it has someting with the buffer size and the
time spending on different threads.Maybe the tiem spend by consumer thread is too long and
the producer has to wait until the next buffer is available.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
|