hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Trivial Update of "Hamburg" by edwardyoon
Date Wed, 26 Aug 2009 01:41:08 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by edwardyoon:

  Each worker will process the data fragments stored locally. And then, We can do bulk synchronization
using collected communication data. The 'Computation' and 'Bulk synchronization' can be performed
iteratively, Data for synchronization can be compressed to reduce network usage. The main
difference between Hamburg and M/R is that Hamburg does not make intermediate data aggregate
into reducer. Instead, each computation node communicates only necessary data into one another.
 It will be efficient if total communicated data is smaller then intermediate data to be aggregated
into reducers. Plainly, It aims to improve the performance of traverse operations in Graph
- For example, let's assume the graph looks like presented below:
- [http://lh5.ggpht.com/_DBxyBGtfa3g/SmQTwhOSGwI/AAAAAAAABmY/ERiJ2BUFxI0/s800/figure2.PNG]
- We can store the graph as a adjacency matrix in HDFS or HBase and use MapReduce to explores
all the neighboring nodes, but we need a lot of iterations to get next vertex per-hop time
since as mentioned motivation. However, practically, we need only one communication between
server2 and server3 for determine the distance between 6 and 7 as below:
- [http://lh6.ggpht.com/_DBxyBGtfa3g/SmQTwvxT2zI/AAAAAAAABmc/BRrv7plzPtc/s800/figure3.PNG]
-  ''TODO: write more detail of above. -- Edward J.''
  === Initial contributors ===
   * Edward J. (edwardyoon AT apache.org)
   * Hyunsik Choi (hyunsik.choi AT gmail.com)
@@ -48, +38 @@

  == Implementation ==
-  * Work in progress
+ The user interface is likely to be a simple form that allows processing a single item at
a time. 
+ {{{
+ User Interface:
+ public void traverse(Map<Vertex, Message> value, Map<Integer, Message> nextQueue)
+   for (Integer i : v.getAdjList()) {
+     // Handle the vertex and send a message with modified state of vertex i to next level
+   }
+ }
+ }}}
  == Related Projects ==

View raw message