incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-443) SSSP doesn't works with multi-tasks
Date Wed, 09 Nov 2011 11:38:51 GMT

    [ https://issues.apache.org/jira/browse/HAMA-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146953#comment-13146953
] 

Thomas Jungblut commented on HAMA-443:
--------------------------------------

This could be a bit more difficult. 

First I have to regenerate the input to a sequencefile which it was before.
Second there is very likely a problem with the partitioning: after partitioning the splits
are inside an array, JobInProgress is going to setup new tasks and assigns the split to the
task. This assignment is NOT consistent with the peername indices in our array of peers (you
know, sorted by name for master tasks). 
Two solutions:
 - Make the partition index visible for the user, then in the first superstep each task has
to broadcast the partition <- not very good..
 - Instead of sorting by name we should sort ascending by the partition index if it is available.
This would mean, that the partition id would be stored in zookeeper as well. <- not a good
solution for us, but I guess it would be more convinient for the user.

More generall sorting by name seems to be not a good strategy in the long run. Consider we
have multiple tasks and get in touch with fault tolerance. A groom handles 3 tasks on port
33, 34, 35. groom:33 is faulty and gets killed. Now it will be respawned, because the port
is blocked it will be groom:36. 
Now the task isn't the master anymore, because the whole sequence is sorted by the name.

In my opinion we should sort ascending by taskId, this would actually solve the problem with
partitioning as well, because the index is the same index in the taskid.
                
> SSSP doesn't works with multi-tasks
> -----------------------------------
>
>                 Key: HAMA-443
>                 URL: https://issues.apache.org/jira/browse/HAMA-443
>             Project: Hama
>          Issue Type: Bug
>          Components: examples
>    Affects Versions: 0.3.0
>            Reporter: Edward J. Yoon
>            Assignee: Thomas Jungblut
>             Fix For: 0.4.0
>
>
> {code}
> root@Cnode1:/usr/local/src/hama-trunk# core/bin/hama jar examples/target/hama-examples-0.4.0-incubating-SNAPSHOT.jar
sssp Klewno xx /user/root/edward/sssp-adjacencylist.txt
> Single Source Shortest Path Example:
> <Startvertex name> <optional: output path> <optional: path to own adjacency
list textfile!>
> Setting default start vertex to "Frankfurt"!
> Setting start vertex to Klewno!
> Using new output folder: xx
> 11/09/26 09:46:24 INFO graph.ShortestPaths: Starting data partitioning...
> 11/09/26 09:47:12 INFO graph.ShortestPaths: Finished!
> 11/09/26 09:47:12 INFO bsp.BSPJobClient: Running job: job_201109260929_0004
> 11/09/26 09:47:15 INFO bsp.BSPJobClient: Current supersteps number: 0
> 11/09/26 09:47:21 INFO bsp.BSPJobClient: Current supersteps number: 1
> 11/09/26 09:47:33 INFO bsp.BSPJobClient: The total number of supersteps: 1
> Job Finished in 21.553 seconds
> -------------------- RESULTS --------------------
> java.lang.NullPointerException
>         at org.apache.hama.examples.graph.ShortestPathsBase.printOutput(ShortestPathsBase.java:93)
>         at org.apache.hama.examples.graph.ShortestPaths.main(ShortestPaths.java:239)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>         at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>         at org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:37)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hama.util.RunJar.main(RunJar.java:145)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message