hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Trivial Update of "Hamburg" by edwardyoon
Date Tue, 21 Jul 2009 02:14:50 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by edwardyoon:
http://wiki.apache.org/hadoop/Hamburg

------------------------------------------------------------------------------
   * Computation on data that reside in local storage; it is similar to map operation in M/R.
   * Each node communicates its necessary data into one another.
   * All processors synchronize which waits for all of the communications actions to complete.
- The main difference between Hamburg and M/R is that Hamburg does not make intermediate data
aggregate into reducer. Instead, each computation node communicates only necessary data into
one another. 
- It will be efficient if total communicated data is smaller then intermediate data to be
aggregated into reducers.
+ 
+ Let's see more detail in the diagram of computing method of Hamburg based on BSP model.
+ 
+ [http://lh4.ggpht.com/_DBxyBGtfa3g/SmQUYTHWooI/AAAAAAAABmk/cFVlLCdLVHE/s800/figure1.PNG]
+ 
+ Each worker will process the data fragments stored locally. And then, We can do bulk synchronization
using collected communication data. The 'Computation' and 'Bulk synchronization' can be performed
iteratively, Data for synchronization can be compressed to reduce network usage. The main
difference between Hamburg and M/R is that Hamburg does not make intermediate data aggregate
into reducer. Instead, each computation node communicates only necessary data into one another.
 It will be efficient if total communicated data is smaller then intermediate data to be aggregated
into reducers. Plainly, It aims to improve the performance of traverse operations in Graph
computing. 
+ 
+ For example, to explores all the neighboring nodes from the root node using Map/Reduce (FYI,
[http://blog.udanax.org/2009/02/breadth-first-search-mapreduce.html Breadth-First Search (BFS)
& MapReduce]), We need a lot of iterations to get next vertex per-hop time.
+ 
+ Let's assume the graph looks like presented below:
+ 
+ [http://lh5.ggpht.com/_DBxyBGtfa3g/SmQTwhOSGwI/AAAAAAAABmY/ERiJ2BUFxI0/s800/figure2.PNG]
  
  === Initial contributors ===
   * Edward J. (edwardyoon AT apache.org)

Mime
View raw message