hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HAMA-642) Make GraphRunner disk based
Date Mon, 24 Sep 2012 04:02:08 GMT

     [ https://issues.apache.org/jira/browse/HAMA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Edward J. Yoon updated HAMA-642:
--------------------------------

    Fix Version/s:     (was: 0.6.0)

Job never finishes. 

I doubt there's a JDBM's bug (concurrent access or structural bug might be a part of the problem)
or bug in this patch.

{code}
[oracle@bda01 hama-0.6.0-MEM]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar sssp 3 /user/oracle/edward/rangraph/part-r-00000
testout
12/09/23 19:20:12 INFO bsp.FileInputFormat: Total input paths to process : 1
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.12:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.14:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.7:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.8:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.3:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.2:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.17:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.18:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node: /switch1/192.168.10.5:50010
12/09/23 19:20:12 INFO bsp.FileInputFormat: Total # of splits: 85
12/09/23 19:20:12 INFO bsp.BSPJobClient: Running job: job_201209231919_0001
12/09/23 19:20:15 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/23 19:20:18 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/23 19:20:27 INFO bsp.BSPJobClient: Current supersteps number: 3
12/09/23 19:20:36 INFO bsp.BSPJobClient: Current supersteps number: 8
12/09/23 19:20:39 INFO bsp.BSPJobClient: Current supersteps number: 10
12/09/23 19:20:42 INFO bsp.BSPJobClient: Current supersteps number: 12
12/09/23 19:20:48 INFO bsp.BSPJobClient: Current supersteps number: 13
12/09/23 19:21:00 INFO bsp.BSPJobClient: Current supersteps number: 14
12/09/23 19:21:06 INFO bsp.BSPJobClient: Current supersteps number: 15
12/09/23 19:21:15 INFO bsp.BSPJobClient: Current supersteps number: 16
12/09/23 19:21:21 INFO bsp.BSPJobClient: Current supersteps number: 17
12/09/23 19:21:30 INFO bsp.BSPJobClient: Current supersteps number: 18
12/09/23 19:21:36 INFO bsp.BSPJobClient: Current supersteps number: 19
12/09/23 19:21:42 INFO bsp.BSPJobClient: Current supersteps number: 20
12/09/23 19:21:48 INFO bsp.BSPJobClient: Current supersteps number: 21
12/09/23 19:21:54 INFO bsp.BSPJobClient: Current supersteps number: 22
12/09/23 19:21:57 INFO bsp.BSPJobClient: Current supersteps number: 23
12/09/23 19:22:03 INFO bsp.BSPJobClient: Current supersteps number: 24
12/09/23 19:22:06 INFO bsp.BSPJobClient: Current supersteps number: 26
12/09/23 19:22:09 INFO bsp.BSPJobClient: Current supersteps number: 28
12/09/23 19:22:12 INFO bsp.BSPJobClient: Current supersteps number: 31
12/09/23 19:22:15 INFO bsp.BSPJobClient: Current supersteps number: 33
12/09/23 19:22:18 INFO bsp.BSPJobClient: Current supersteps number: 37
12/09/23 19:22:21 INFO bsp.BSPJobClient: Current supersteps number: 40
12/09/23 19:22:24 INFO bsp.BSPJobClient: Current supersteps number: 48
12/09/23 19:22:27 INFO bsp.BSPJobClient: Current supersteps number: 49
12/09/23 19:22:27 INFO bsp.BSPJobClient: The total number of supersteps: 49
12/09/23 19:22:27 INFO bsp.BSPJobClient: Counters: 8
12/09/23 19:22:27 INFO bsp.BSPJobClient:   org.apache.hama.bsp.JobInProgress$JobCounter
12/09/23 19:22:27 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=85
12/09/23 19:22:27 INFO bsp.BSPJobClient:   org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/23 19:22:27 INFO bsp.BSPJobClient:     SUPERSTEPS=49
12/09/23 19:22:27 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=4165
12/09/23 19:22:27 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=1692428
12/09/23 19:22:27 INFO bsp.BSPJobClient:     IO_BYTES_READ=1167771205
12/09/23 19:22:27 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_SENT=1157989750
12/09/23 19:22:27 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=10000000
12/09/23 19:22:27 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_RECEIVED=578994875
Job Finished in 135.463 seconds

----

[oracle@bda01 hama-0.6.0-JDBM]$ bin/hama jar ../hama-0.6.0-MEM/hama-examples-0.6.0-SNAPSHOT.jar
sssp 3 /user/oracle/edward/rangraph/part-r-00000 testout
....
12/09/23 20:43:48 INFO bsp.BSPJobClient: Current supersteps number: 485
12/09/23 20:44:00 INFO bsp.BSPJobClient: Current supersteps number: 486
12/09/23 20:44:09 INFO bsp.BSPJobClient: Current supersteps number: 487
12/09/23 20:44:18 INFO bsp.BSPJobClient: Current supersteps number: 488
12/09/23 20:44:30 INFO bsp.BSPJobClient: Current supersteps number: 489
12/09/23 20:44:39 INFO bsp.BSPJobClient: Current supersteps number: 490
12/09/23 20:44:48 INFO bsp.BSPJobClient: Current supersteps number: 491
12/09/23 20:45:00 INFO bsp.BSPJobClient: Current supersteps number: 492
12/09/23 20:45:09 INFO bsp.BSPJobClient: Current supersteps number: 493
12/09/23 20:45:18 INFO bsp.BSPJobClient: Current supersteps number: 494
12/09/23 20:45:30 INFO bsp.BSPJobClient: Current supersteps number: 495
12/09/23 20:45:39 INFO bsp.BSPJobClient: Current supersteps number: 496
12/09/23 20:45:48 INFO bsp.BSPJobClient: Current supersteps number: 497
12/09/23 20:46:00 INFO bsp.BSPJobClient: Current supersteps number: 498
12/09/23 20:46:09 INFO bsp.BSPJobClient: Current supersteps number: 499
12/09/23 20:46:18 INFO bsp.BSPJobClient: Current supersteps number: 500
12/09/23 20:46:30 INFO bsp.BSPJobClient: Current supersteps number: 501
^C[oracle@bda01 hama-0.6.0-JDBM]$ 
[oracle@bda01 hama-0.6.0-JDBM]$ bin/hama job -list all
1 jobs submitted
States are:
	Running : 1	Succeded : 2	Failed : 3	Prep : 4
JobId	State	StartTime	UserName
job_201209231923_0001	1	1348453426073	oracle
[oracle@bda01 hama-0.6.0-JDBM]$ bin/hama job -kill job_201209231923_0001
Killed job job_201209231923_0001
{code}
                
> Make GraphRunner disk based
> ---------------------------
>
>                 Key: HAMA-642
>                 URL: https://issues.apache.org/jira/browse/HAMA-642
>             Project: Hama
>          Issue Type: Improvement
>          Components: graph
>    Affects Versions: 0.5.0
>            Reporter: Thomas Jungblut
>            Assignee: Edward J. Yoon
>         Attachments: HAMA-642_unix_1.patch, HAMA-642_unix_2.patch, HAMA-scale_1.patch,
HAMA-scale_2.patch, HAMA-scale_3.patch, HAMA-scale_4.patch
>
>
> To improve scalability we can improve the graph runner to be disk based.
> Which basically means:
> - We have just a single Vertex instance that get's refilled.
> - We directly write vertices to disk after partitioning
> - In every superstep we iterate over the vertices on disk, fill the vertex instance and
call the users compute functions
> Problems:
> - State other than vertex value can't be stored easy
> - How do we deal with random access after messages have arrived?
> So I think we should make the graph runner more hybrid, like using the queues we have
implemented in the messaging. So the graphrunner can be configured to run completely on disk,
in cached mode or in in-memory mode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message