incubator-hama-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hama Wiki] Trivial Update of "Architecture" by edwardyoon
Date Wed, 20 Oct 2010 05:21:26 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hama Wiki" for change notification.

The "Architecture" page has been changed by edwardyoon.
http://wiki.apache.org/hama/Architecture?action=diff&rev1=138&rev2=139

--------------------------------------------------

  <<TableOfContents(5)>>
- 
- This effort is still a "work in progress". Please feel free to add comments.
- ----
  
  == Introduction ==
  
- HAMA is a distributed computing framework based on BSP computing techniques for massive
scientific computations (e.g., matrix, graph, network, ..., etc), 
+ HAMA is a distributed computing framework based on BSP computing techniques for massive
scientific computations (e.g., matrix, graph, network, ..., etc), Currently being incubated
as one of the incubator project by the Apache Software Foundation.
- currently being incubated as one of the incubator project by the Apache Software Foundation.
We are focusing on are as follows:
- 
-  * Compatibility with other distributed storage system (Hadoop HDFS, Hbase, CouchDB, ..,
etc)
-  * Flexibility and Scalability of BSP applications
  
  == The overall architecture of HAMA ==
  
- Hama is consist of BSP, a network-based distributed computing engine, and a set of applications.
+ Hama is consists of three major components: BSPMaster, GroomServers and Zookeeper. The below
diagram displays a overview of the overall architecture and shows which components communicate
with each other. The components will be explained in detail in the course of this document.
  
- === BSP framework ===
+ === BSPMaster ===
  
- Hama BSP is a network-based distributed computing engine. The idea of network-based distributed
computing using commodity PCs, and The BSP (Bulk Synchronous Parallel) model is not new. See
http://en.wikipedia.org/wiki/Bulk_synchronous_parallel more detailed information of BSP. We
are inspired from Google Pregel, we believe that the BSP programming paradigm is ideally suited
for complex problems requiring communication of data, such as matrix, graph.
+ BSPMaster is responsible to do the following:
  
- One BSP cluster consists of one BSPMaster, multiple !GroomServer and one or more zookeeper
servers in a network environment.
- 
- ==== BSPMaster ====
- BSPMaster is responsible to do the following: 
-  1. Maintaining groom server status. 
+  * Maintaining groom server status. 
-  2. Controlling super steps in a cluster. 
+  * Controlling super steps in a cluster. 
-  3. Maintaining job progress information. 
+  * Maintaining job progress information. 
-  4. Scheduling Jobs and Assigning tasks to groom servers 
+  * Scheduling Jobs and Assigning tasks to groom servers 
-  5. Disseminating execution class across groom servers.
+  * Disseminating execution class across groom servers.
-  6. Controlling fault. 
+  * Controlling fault. 
-  7. Providing users with the cluster control interface.
+  * Providing users with the cluster control interface.
  
  A BSP Master and multiple grooms are started by the script. Then, the bsp master starts
up with a RPC server for groom servers. Groom servers starts up with a BSPPeer instance -
later, BSPPeer needs to be integrated with !GroomServer - and a RPC proxy to contact the bsp
master. After started, each groom periodically sends a heartbeat message that encloses its
groom server status, including maximum task capacity, unused memory, and so on. 
  
  Each time the bsp master receives a heartbeat message, it brings up-to-date groom server
status - the bsp master makes use of groom servers' status in order to effectively assign
tasks to idle groom servers - and returns a heartbeat response that contains assigned tasks
and others actions that a groom server has to do. For now, we have a FIFO job scheduler and
very simple task assignment algorithms.
+ 
- ==== GroomServer ====
+ === GroomServer ===
+ 
  A Groom Server (shortly referred to as groom) is a process that performs bsp tasks assigned
by BSPMaster. Each groom contacts the BSPMaster, and it takes assigned tasks and reports its
status by means of periodical piggybacks with BSPMaster. Each groom is designed to run with
HDFS or other distributed storages. Basically, a groom server and a data node should be run
on one physical node.
  
- ==== BSP Programming Model ====
+ === Zookeeper ===
  
- Hama BSP programming model is designed to be similar to the !MapReduce programming model.
This allows the framework to be less complex than the traditional BSP libraries capable of
executing sequential code, we to provide user-friendly and intuitive programming interface
to existing users of !MapReduce framework. 
+ A Zookeeper is used to manage the efficient barrier synchronisation of the BSPPeers. Later,
it will also be used for the area of a fault tolerance system.
  
- Let's see the below example code to understand it:
+ == BSP Programming Model ==
+ 
+ A BSP program consists of a sequence of supersteps. Each superstep consists of the three
phases below:
+ 
+  * Local computation
+  * Process communication
+  * Barrier synchronization
+ 
+ And, each of these phases will be defined in a bsp() method by user. Let's see the below
example code:
  
  {{{
  public class BSPEaxmple {
@@ -53, +52 @@

      @Override
      public void bsp(BSPPeer bspPeer) throws IOException, KeeperException,
          InterruptedException {
-       // A communication and synchronization phase of a BSP superstep
+       // 1. Do something locally
        
-       // Send data to neighbor node
+       // 2. Send data to neighbor node
        bspPeer.send(hostname, msg);
+       // Or, receive messages
+       while ((message = bspPeer.getCurrentMessage()) != null) {
+          byte[] data = message.getData();
+       }
  
-       // Superstep synchronization
+       // 3. Barrier synchronization
        bspPeer.sync();
- 
-       // Receive current messages
-       bspPeer.getCurrentMessage();
      }
  
      @Override
@@ -87, +87 @@

      // Submit job
      BSPJobClient.runJob(bsp);
    }
- 
  }  
  }}}
  
- == An Examples ==
+ == An BSP examples ==
  
   * [[SerializePrinting|Serialize Printing Of "Hello BSP"]]
   * [[PiEstimator|BSP based Pi Estimator]]
  
- == What's next? ==
- 
-  * [[RoadMap|Hama Project Roadmap]]
- 

Mime
View raw message