hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-598) Destination of Edge should be set according to partitionID with or without runtime partitioning option.
Date Wed, 27 Jun 2012 08:01:47 GMT

    [ https://issues.apache.org/jira/browse/HAMA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402045#comment-13402045
] 

Edward J. Yoon commented on HAMA-598:
-------------------------------------

Quick commit for the test, but beautiful solve of this problem is requires big refactoring.

{code}
Index: graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java
===================================================================
--- graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java	(revision 1354305)
+++ graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java	(working copy)
@@ -391,16 +391,16 @@
     Vertex<V, E, M> vertex = newVertexInstance(vertexClass, conf);
     vertex.setPeer(peer);
     vertex.runner = this;
-    while (true) {
-      KeyValuePair<Writable, Writable> next = peer.readNext();
-      if (next == null) {
-        break;
-      }
+    
+    KeyValuePair<Writable, Writable> next = null;
+    int lines = 0;
+    while ((next = peer.readNext()) != null) {
       boolean vertexFinished = reader.parseVertex(next.getKey(),
           next.getValue(), vertex);
       if (!vertexFinished) {
         continue;
       }
+      
       if (vertex.getEdges() == null) {
         vertex.setEdges(new ArrayList<Edge<V, E>>(0));
       }
@@ -420,12 +420,26 @@
         }
         peer.send(peer.getPeerName(partition), new GraphJobMessage(vertex));
       } else {
+        // FIXME need to set destination names
         vertex.setup(conf);
         vertices.put(vertex.getVertexID(), vertex);
       }
       vertex = newVertexInstance(vertexClass, conf);
       vertex.setPeer(peer);
       vertex.runner = this;
+      
+      lines++;
+      if((lines % 50000) == 0) {
+        peer.sync();
+        GraphJobMessage msg = null;
+        while ((msg = peer.getCurrentMessage()) != null) {
+          Vertex<V, E, M> messagedVertex = (Vertex<V, E, M>) msg.getVertex();
+          messagedVertex.setPeer(peer);
+          messagedVertex.runner = this;
+          messagedVertex.setup(conf);
+          vertices.put(messagedVertex.getVertexID(), messagedVertex);
+        }
+      }
     }
 
     if (runtimePartitioning) {
@@ -440,6 +454,8 @@
       }
     }
 
+    LOG.info("Loading finished at " + peer.getSuperstepCount() + " steps.");
+    
     /*
      * If the user want to repair the graph, it should traverse through that
      * local chunk of adjancency list and message the corresponding peer to
{code}
                
> Destination of Edge should be set according to partitionID with or without runtime partitioning
option.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HAMA-598
>                 URL: https://issues.apache.org/jira/browse/HAMA-598
>             Project: Hama
>          Issue Type: Bug
>          Components: graph
>    Affects Versions: 0.5.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>            Priority: Critical
>             Fix For: 0.5.0
>
>         Attachments: HAMA-598.patch
>
>
> Otherwise, you'll see NullPointerExceptions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message