hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Patterson <j...@cloudera.com>
Subject Re: HDFS Source Code
Date Mon, 07 Jun 2010 19:47:25 GMT
A nice overview of Map Reduce is at:


<http://hadoop.apache.org/common/docs/current/mapred_tutorial.html>In terms
of replacing the entire communication framework under Map Reduce, thats a
considerably different and more complex task that simply talking to hdfs
from java. You have a large number of interlocking classes based on the
communication system of hadoop; replacing this would be like trying to
replace the frame of an automobile --- in other words, you might try, but
you run a high risk of not having a large percentage of the parts not work
correctly with the new automobile frame.

If you were to tackle such a large task, it would be less of "knowing a
certain set of classes to work with" and more of knowing how a large degree
of hadoop works.


On Mon, Jun 7, 2010 at 12:20 PM, Ahmad Shahzad <ashahzad4@gmail.com> wrote:

> Hi,
>    I am also interested in looking what is going on under the hood. Thanks
> for sharing this information.
> Could you please tell the same about Map Reduce. I mean, how to figure out
> that how job tracker and task trackers communicate with each other. Which
> classes in hadoop api are responsible for doing that. If i go in detail, i
> would like to know that how can i change the communication mechanism of
> hadoop map reduce to use my communication library instead of using regular
> sockets and http.
> Regards,
> Ahmad Shahzad

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message