hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [jira] [Commented] (HAMA-359) Development of Shortest Path Finding Algorithm
Date Wed, 18 May 2011 09:18:57 GMT
FYI,

Below is the test result on 16 nodes (16 * 16 cores)

----
root@Cnode1:/usr/local/src/hama-trunk# bin/hama jar
hama-0.3.0-examples.jar sssp Klewno /user/root/output /user/root/d
Single Source Shortest Path Example:
<Startvertex name> <optional: output path> <optional: path to own adjacency
list
Setting default start vertex to "Frankfurt"!
Setting start vertex to Klewno!
Using new output folder: /user/root/output
11/05/18 16:44:23 INFO examples.ShortestPaths: Starting data partitioning...
11/05/18 16:44:23 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new decompressor
11/05/18 16:51:38 INFO examples.ShortestPaths: Finished!
11/05/18 16:51:38 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
hdfs://hnode15:9000/tmp/hadoop-root/bsp/system/su
11/05/18 16:51:38 INFO bsp.BSPJobClient: Running job: job_201105181515_0002
11/05/18 16:51:41 INFO bsp.BSPJobClient: Current supersteps number: 0
11/05/18 16:52:02 INFO bsp.BSPJobClient: Current supersteps number: 1
11/05/18 16:52:11 INFO bsp.BSPJobClient: Current supersteps number: 2
11/05/18 16:52:20 INFO bsp.BSPJobClient: Current supersteps number: 3
11/05/18 16:52:32 INFO bsp.BSPJobClient: Current supersteps number: 4
11/05/18 16:52:41 INFO bsp.BSPJobClient: Current supersteps number: 5
11/05/18 16:52:53 INFO bsp.BSPJobClient: Current supersteps number: 6
11/05/18 16:53:02 INFO bsp.BSPJobClient: Current supersteps number: 7
11/05/18 16:53:14 INFO bsp.BSPJobClient: Current supersteps number: 8
11/05/18 16:53:23 INFO bsp.BSPJobClient: Current supersteps number: 9
11/05/18 16:53:35 INFO bsp.BSPJobClient: Current supersteps number: 10
11/05/18 16:53:44 INFO bsp.BSPJobClient: Current supersteps number: 11
11/05/18 16:53:56 INFO bsp.BSPJobClient: Current supersteps number: 12
11/05/18 16:54:08 INFO bsp.BSPJobClient: Current supersteps number: 13
11/05/18 16:54:20 INFO bsp.BSPJobClient: Current supersteps number: 14
11/05/18 16:54:33 INFO bsp.BSPJobClient: Current supersteps number: 15
11/05/18 16:55:18 INFO bsp.BSPJobClient: Current supersteps number: 16
11/05/18 16:55:39 INFO bsp.BSPJobClient: Current supersteps number: 17
11/05/18 16:55:51 INFO bsp.BSPJobClient: Current supersteps number: 18
11/05/18 17:01:33 INFO bsp.BSPJobClient: Current supersteps number: 19
11/05/18 17:03:51 INFO bsp.BSPJobClient: Current supersteps number: 20
11/05/18 17:04:00 INFO bsp.BSPJobClient: Current supersteps number: 21
11/05/18 17:23:09 INFO bsp.BSPJobClient: Current supersteps number: 22
11/05/18 17:30:12 INFO bsp.BSPJobClient: Current supersteps number: 23
11/05/18 17:30:24 INFO bsp.BSPJobClient: Current supersteps number: 24
11/05/18 17:40:01 INFO bsp.BSPJobClient: Current supersteps number: 25
11/05/18 17:43:43 INFO bsp.BSPJobClient: Current supersteps number: 26
11/05/18 17:43:52 INFO bsp.BSPJobClient: Current supersteps number: 27
11/05/18 17:47:25 INFO bsp.BSPJobClient: Current supersteps number: 28
11/05/18 17:48:52 INFO bsp.BSPJobClient: Current supersteps number: 29
11/05/18 17:49:04 INFO bsp.BSPJobClient: Current supersteps number: 30
11/05/18 17:50:28 INFO bsp.BSPJobClient: Current supersteps number: 31
11/05/18 17:51:07 INFO bsp.BSPJobClient: Current supersteps number: 32
11/05/18 17:51:16 INFO bsp.BSPJobClient: Current supersteps number: 33
11/05/18 17:51:52 INFO bsp.BSPJobClient: Current supersteps number: 34
11/05/18 17:52:13 INFO bsp.BSPJobClient: Current supersteps number: 35
11/05/18 17:52:22 INFO bsp.BSPJobClient: Current supersteps number: 36
11/05/18 17:52:40 INFO bsp.BSPJobClient: Current supersteps number: 37
11/05/18 17:52:55 INFO bsp.BSPJobClient: Current supersteps number: 38
11/05/18 17:53:04 INFO bsp.BSPJobClient: Current supersteps number: 39
11/05/18 17:53:16 INFO bsp.BSPJobClient: Current supersteps number: 40
11/05/18 17:53:28 INFO bsp.BSPJobClient: Current supersteps number: 41
11/05/18 17:53:37 INFO bsp.BSPJobClient: Current supersteps number: 42
11/05/18 17:53:49 INFO bsp.BSPJobClient: Current supersteps number: 43
11/05/18 17:54:01 INFO bsp.BSPJobClient: Current supersteps number: 44
11/05/18 17:54:10 INFO bsp.BSPJobClient: Current supersteps number: 45
11/05/18 17:54:22 INFO bsp.BSPJobClient: Current supersteps number: 46
11/05/18 17:54:31 INFO bsp.BSPJobClient: Current supersteps number: 47
11/05/18 17:54:43 INFO bsp.BSPJobClient: Current supersteps number: 48
11/05/18 17:54:52 INFO bsp.BSPJobClient: Current supersteps number: 49
11/05/18 17:55:01 INFO bsp.BSPJobClient: Current supersteps number: 50
11/05/18 17:55:13 INFO bsp.BSPJobClient: Current supersteps number: 51
11/05/18 17:55:25 INFO bsp.BSPJobClient: Current supersteps number: 52
11/05/18 17:55:34 INFO bsp.BSPJobClient: Current supersteps number: 53
11/05/18 17:55:43 INFO bsp.BSPJobClient: Current supersteps number: 54
11/05/18 17:55:55 INFO bsp.BSPJobClient: Current supersteps number: 55
11/05/18 17:56:04 INFO bsp.BSPJobClient: Current supersteps number: 56
11/05/18 17:56:16 INFO bsp.BSPJobClient: Current supersteps number: 57
11/05/18 17:56:25 INFO bsp.BSPJobClient: Current supersteps number: 58
11/05/18 17:56:37 INFO bsp.BSPJobClient: Current supersteps number: 59
11/05/18 17:56:46 INFO bsp.BSPJobClient: Current supersteps number: 60
11/05/18 17:56:58 INFO bsp.BSPJobClient: Current supersteps number: 61
11/05/18 17:57:07 INFO bsp.BSPJobClient: Current supersteps number: 62
11/05/18 17:57:19 INFO bsp.BSPJobClient: Current supersteps number: 63
11/05/18 17:57:28 INFO bsp.BSPJobClient: Current supersteps number: 64
11/05/18 17:57:37 INFO bsp.BSPJobClient: Current supersteps number: 65
11/05/18 17:57:49 INFO bsp.BSPJobClient: Current supersteps number: 66
11/05/18 17:57:58 INFO bsp.BSPJobClient: Current supersteps number: 67
11/05/18 17:58:10 INFO bsp.BSPJobClient: Current supersteps number: 68
11/05/18 17:58:19 INFO bsp.BSPJobClient: Current supersteps number: 69
11/05/18 17:58:25 INFO bsp.BSPJobClient: The total number of supersteps: 69
Job Finished in 4006.779 seconds
-------------------- RESULTS --------------------
java.io.IOException: Cannot open filename /user/root/output/_logs
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1497)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1488)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:376)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178)
        at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1437)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
        at org.apache.hama.examples.ShortestPaths.printOutput(ShortestPaths.java:299)
        at org.apache.hama.examples.ShortestPaths.main(ShortestPaths.java:550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hama.util.RunJar.main(RunJar.java:145)


On Wed, May 18, 2011 at 5:04 PM, Edward J. Yoon (JIRA) <jira@apache.org> wrote:
>
>    [ https://issues.apache.org/jira/browse/HAMA-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035230#comment-13035230
]
>
> Edward J. Yoon commented on HAMA-359:
> -------------------------------------
>
> I tested after changing 'Thread.sleep(100);' to 'Thread.sleep(10000);' at BSPPeer.sync()
method and finally, my job successfully done.
>
> {code}
>
> 2011-05-18 15:15:27,147 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 40000:
starting
> 2011-05-18 15:15:27,149 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 40000:
starting
> 2011-05-18 15:15:27,151 INFO org.apache.hama.bsp.BSPMaster: Starting RUNNING
> 2011-05-18 15:22:48,062 DEBUG org.apache.hama.bsp.JobInProgress: numBSPTasks: 16
> 2011-05-18 15:22:48,065 DEBUG org.apache.hama.bsp.JobInProgress: Job is initialized.
> 2011-05-18 16:29:27,582 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000005_0'
has finished successfully.
> 2011-05-18 16:29:27,583 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000005'
has completed.
> 2011-05-18 16:29:27,806 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000004_0'
has finished successfully.
> 2011-05-18 16:29:27,806 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000004'
has completed.
> 2011-05-18 16:29:28,336 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000015_0'
has finished successfully.
> 2011-05-18 16:29:28,336 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000015'
has completed.
> 2011-05-18 16:29:28,517 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000000_0'
has finished successfully.
> 2011-05-18 16:29:28,517 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000000'
has completed.
> 2011-05-18 16:29:28,524 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000013_0'
has finished successfully.
> 2011-05-18 16:29:28,524 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000013'
has completed.
> 2011-05-18 16:29:28,589 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000012_0'
has finished successfully.
> 2011-05-18 16:29:28,589 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000012'
has completed.
> 2011-05-18 16:29:28,602 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000001_0'
has finished successfully.
> 2011-05-18 16:29:28,602 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000001'
has completed.
> 2011-05-18 16:29:28,775 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000014_0'
has finished successfully.
> 2011-05-18 16:29:28,775 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000014'
has completed.
> 2011-05-18 16:29:28,909 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000010_0'
has finished successfully.
> 2011-05-18 16:29:28,909 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000010'
has completed.
> 2011-05-18 16:29:28,914 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000007_0'
has finished successfully.
> 2011-05-18 16:29:28,914 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000007'
has completed.
> 2011-05-18 16:29:28,960 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000006_0'
has finished successfully.
> 2011-05-18 16:29:28,960 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000006'
has completed.
> 2011-05-18 16:29:29,148 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000011_0'
has finished successfully.
> 2011-05-18 16:29:29,148 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000011'
has completed.
> 2011-05-18 16:29:29,199 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000008_0'
has finished successfully.
> 2011-05-18 16:29:29,199 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000008'
has completed.
> 2011-05-18 16:29:29,244 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000009_0'
has finished successfully.
> 2011-05-18 16:29:29,244 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000009'
has completed.
> 2011-05-18 16:29:29,274 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000002_0'
has finished successfully.
> 2011-05-18 16:29:29,274 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000002'
has completed.
> 2011-05-18 16:29:29,392 INFO org.apache.hama.bsp.JobInProgress: Taskid 'attempt_201105181515_0001_000003_0'
has finished successfully.
> 2011-05-18 16:29:29,392 INFO org.apache.hama.bsp.TaskInProgress: Task 'task_201105181515_0001_000003'
has completed.
> 2011-05-18 16:29:29,395 DEBUG org.apache.hama.bsp.JobInProgress: Job successfully done.
> 2011-05-18 16:32:48,365 DEBUG org.apache.hama.bsp.BSPMaster: returns all jobs: 1
> {code}
>
>>> Is this still related to the barrier sync?
>
> Yes. the problem related with zk node creation/deletion logic in enterBarrier() and leaveBarrier()
methods. Sometimes they occurs at the same time.
>
>>> Increasing the timeout won't fix the problem with it.
>
> As i mentioned on chat, JVM garbage collection pause causes zk session time-out errors.
>
>> Development of Shortest Path Finding Algorithm
>> ----------------------------------------------
>>
>>                 Key: HAMA-359
>>                 URL: https://issues.apache.org/jira/browse/HAMA-359
>>             Project: Hama
>>          Issue Type: New Feature
>>          Components: examples
>>    Affects Versions: 0.2.0
>>            Reporter: Edward J. Yoon
>>            Assignee: Thomas Jungblut
>>              Labels: gsoc, gsoc2011, mentor
>>             Fix For: 0.3.0
>>
>>         Attachments: HAMA-359-v2.patch, HAMA-359-v3.patch, HAMA-359-v4.patch,
HAMA-359.patch, eddie.patch
>>
>>   Original Estimate: 2016h
>>  Remaining Estimate: 2016h
>>
>> The goal of this project is development of parallel algorithm for finding a Shortest
Path using Hama BSP.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>



-- 
Best Regards, Edward J. Yoon

Mime
View raw message