incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@googlemail.com>
Subject Re: [jira] [Commented] (HAMA-359) Development of Shortest Path Finding Algorithm
Date Wed, 18 May 2011 09:22:50 GMT
Funny how log outputs coming into this directory. There shouldn't be any
logs. I create a filter for that later on.

Thanks for the testing Edward.

2011/5/18 Edward J. Yoon <edwardyoon@apache.org>

> FYI,
>
> Below is the test result on 16 nodes (16 * 16 cores)
>
> ----
> root@Cnode1:/usr/local/src/hama-trunk# bin/hama jar
> hama-0.3.0-examples.jar sssp Klewno /user/root/output /user/root/d
> Single Source Shortest Path Example:
> <Startvertex name> <optional: output path> <optional: path to own adjacency
> list
> Setting default start vertex to "Frankfurt"!
> Setting start vertex to Klewno!
> Using new output folder: /user/root/output
> 11/05/18 16:44:23 INFO examples.ShortestPaths: Starting data
> partitioning...
> 11/05/18 16:44:23 WARN util.NativeCodeLoader: Unable to load native-hadoop
> libra
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor
> 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new decompressor
> 11/05/18 16:51:38 INFO examples.ShortestPaths: Finished!
> 11/05/18 16:51:38 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
> hdfs://hnode15:9000/tmp/hadoop-root/bsp/system/su
> 11/05/18 16:51:38 INFO bsp.BSPJobClient: Running job: job_201105181515_0002
> 11/05/18 16:51:41 INFO bsp.BSPJobClient: Current supersteps number: 0
> 11/05/18 16:52:02 INFO bsp.BSPJobClient: Current supersteps number: 1
> 11/05/18 16:52:11 INFO bsp.BSPJobClient: Current supersteps number: 2
> 11/05/18 16:52:20 INFO bsp.BSPJobClient: Current supersteps number: 3
> 11/05/18 16:52:32 INFO bsp.BSPJobClient: Current supersteps number: 4
> 11/05/18 16:52:41 INFO bsp.BSPJobClient: Current supersteps number: 5
> 11/05/18 16:52:53 INFO bsp.BSPJobClient: Current supersteps number: 6
> 11/05/18 16:53:02 INFO bsp.BSPJobClient: Current supersteps number: 7
> 11/05/18 16:53:14 INFO bsp.BSPJobClient: Current supersteps number: 8
> 11/05/18 16:53:23 INFO bsp.BSPJobClient: Current supersteps number: 9
> 11/05/18 16:53:35 INFO bsp.BSPJobClient: Current supersteps number: 10
> 11/05/18 16:53:44 INFO bsp.BSPJobClient: Current supersteps number: 11
> 11/05/18 16:53:56 INFO bsp.BSPJobClient: Current supersteps number: 12
> 11/05/18 16:54:08 INFO bsp.BSPJobClient: Current supersteps number: 13
> 11/05/18 16:54:20 INFO bsp.BSPJobClient: Current supersteps number: 14
> 11/05/18 16:54:33 INFO bsp.BSPJobClient: Current supersteps number: 15
> 11/05/18 16:55:18 INFO bsp.BSPJobClient: Current supersteps number: 16
> 11/05/18 16:55:39 INFO bsp.BSPJobClient: Current supersteps number: 17
> 11/05/18 16:55:51 INFO bsp.BSPJobClient: Current supersteps number: 18
> 11/05/18 17:01:33 INFO bsp.BSPJobClient: Current supersteps number: 19
> 11/05/18 17:03:51 INFO bsp.BSPJobClient: Current supersteps number: 20
> 11/05/18 17:04:00 INFO bsp.BSPJobClient: Current supersteps number: 21
> 11/05/18 17:23:09 INFO bsp.BSPJobClient: Current supersteps number: 22
> 11/05/18 17:30:12 INFO bsp.BSPJobClient: Current supersteps number: 23
> 11/05/18 17:30:24 INFO bsp.BSPJobClient: Current supersteps number: 24
> 11/05/18 17:40:01 INFO bsp.BSPJobClient: Current supersteps number: 25
> 11/05/18 17:43:43 INFO bsp.BSPJobClient: Current supersteps number: 26
> 11/05/18 17:43:52 INFO bsp.BSPJobClient: Current supersteps number: 27
> 11/05/18 17:47:25 INFO bsp.BSPJobClient: Current supersteps number: 28
> 11/05/18 17:48:52 INFO bsp.BSPJobClient: Current supersteps number: 29
> 11/05/18 17:49:04 INFO bsp.BSPJobClient: Current supersteps number: 30
> 11/05/18 17:50:28 INFO bsp.BSPJobClient: Current supersteps number: 31
> 11/05/18 17:51:07 INFO bsp.BSPJobClient: Current supersteps number: 32
> 11/05/18 17:51:16 INFO bsp.BSPJobClient: Current supersteps number: 33
> 11/05/18 17:51:52 INFO bsp.BSPJobClient: Current supersteps number: 34
> 11/05/18 17:52:13 INFO bsp.BSPJobClient: Current supersteps number: 35
> 11/05/18 17:52:22 INFO bsp.BSPJobClient: Current supersteps number: 36
> 11/05/18 17:52:40 INFO bsp.BSPJobClient: Current supersteps number: 37
> 11/05/18 17:52:55 INFO bsp.BSPJobClient: Current supersteps number: 38
> 11/05/18 17:53:04 INFO bsp.BSPJobClient: Current supersteps number: 39
> 11/05/18 17:53:16 INFO bsp.BSPJobClient: Current supersteps number: 40
> 11/05/18 17:53:28 INFO bsp.BSPJobClient: Current supersteps number: 41
> 11/05/18 17:53:37 INFO bsp.BSPJobClient: Current supersteps number: 42
> 11/05/18 17:53:49 INFO bsp.BSPJobClient: Current supersteps number: 43
> 11/05/18 17:54:01 INFO bsp.BSPJobClient: Current supersteps number: 44
> 11/05/18 17:54:10 INFO bsp.BSPJobClient: Current supersteps number: 45
> 11/05/18 17:54:22 INFO bsp.BSPJobClient: Current supersteps number: 46
> 11/05/18 17:54:31 INFO bsp.BSPJobClient: Current supersteps number: 47
> 11/05/18 17:54:43 INFO bsp.BSPJobClient: Current supersteps number: 48
> 11/05/18 17:54:52 INFO bsp.BSPJobClient: Current supersteps number: 49
> 11/05/18 17:55:01 INFO bsp.BSPJobClient: Current supersteps number: 50
> 11/05/18 17:55:13 INFO bsp.BSPJobClient: Current supersteps number: 51
> 11/05/18 17:55:25 INFO bsp.BSPJobClient: Current supersteps number: 52
> 11/05/18 17:55:34 INFO bsp.BSPJobClient: Current supersteps number: 53
> 11/05/18 17:55:43 INFO bsp.BSPJobClient: Current supersteps number: 54
> 11/05/18 17:55:55 INFO bsp.BSPJobClient: Current supersteps number: 55
> 11/05/18 17:56:04 INFO bsp.BSPJobClient: Current supersteps number: 56
> 11/05/18 17:56:16 INFO bsp.BSPJobClient: Current supersteps number: 57
> 11/05/18 17:56:25 INFO bsp.BSPJobClient: Current supersteps number: 58
> 11/05/18 17:56:37 INFO bsp.BSPJobClient: Current supersteps number: 59
> 11/05/18 17:56:46 INFO bsp.BSPJobClient: Current supersteps number: 60
> 11/05/18 17:56:58 INFO bsp.BSPJobClient: Current supersteps number: 61
> 11/05/18 17:57:07 INFO bsp.BSPJobClient: Current supersteps number: 62
> 11/05/18 17:57:19 INFO bsp.BSPJobClient: Current supersteps number: 63
> 11/05/18 17:57:28 INFO bsp.BSPJobClient: Current supersteps number: 64
> 11/05/18 17:57:37 INFO bsp.BSPJobClient: Current supersteps number: 65
> 11/05/18 17:57:49 INFO bsp.BSPJobClient: Current supersteps number: 66
> 11/05/18 17:57:58 INFO bsp.BSPJobClient: Current supersteps number: 67
> 11/05/18 17:58:10 INFO bsp.BSPJobClient: Current supersteps number: 68
> 11/05/18 17:58:19 INFO bsp.BSPJobClient: Current supersteps number: 69
> 11/05/18 17:58:25 INFO bsp.BSPJobClient: The total number of supersteps: 69
> Job Finished in 4006.779 seconds
> -------------------- RESULTS --------------------
> java.io.IOException: Cannot open filename /user/root/output/_logs
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1497)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1488)
>        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:376)
>        at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1437)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hama.examples.ShortestPaths.printOutput(ShortestPaths.java:299)
>        at
> org.apache.hama.examples.ShortestPaths.main(ShortestPaths.java:550)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>        at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>        at
> org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:34)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hama.util.RunJar.main(RunJar.java:145)
>
>
> On Wed, May 18, 2011 at 5:04 PM, Edward J. Yoon (JIRA) <jira@apache.org>
> wrote:
> >
> >    [
> https://issues.apache.org/jira/browse/HAMA-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035230#comment-13035230]
> >
> > Edward J. Yoon commented on HAMA-359:
> > -------------------------------------
> >
> > I tested after changing 'Thread.sleep(100);' to 'Thread.sleep(10000);' at
> BSPPeer.sync() method and finally, my job successfully done.
> >
> > {code}
> >
> > 2011-05-18 15:15:27,147 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 40000: starting
> > 2011-05-18 15:15:27,149 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 40000: starting
> > 2011-05-18 15:15:27,151 INFO org.apache.hama.bsp.BSPMaster: Starting
> RUNNING
> > 2011-05-18 15:22:48,062 DEBUG org.apache.hama.bsp.JobInProgress:
> numBSPTasks: 16
> > 2011-05-18 15:22:48,065 DEBUG org.apache.hama.bsp.JobInProgress: Job is
> initialized.
> > 2011-05-18 16:29:27,582 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000005_0' has finished successfully.
> > 2011-05-18 16:29:27,583 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000005' has completed.
> > 2011-05-18 16:29:27,806 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000004_0' has finished successfully.
> > 2011-05-18 16:29:27,806 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000004' has completed.
> > 2011-05-18 16:29:28,336 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000015_0' has finished successfully.
> > 2011-05-18 16:29:28,336 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000015' has completed.
> > 2011-05-18 16:29:28,517 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000000_0' has finished successfully.
> > 2011-05-18 16:29:28,517 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000000' has completed.
> > 2011-05-18 16:29:28,524 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000013_0' has finished successfully.
> > 2011-05-18 16:29:28,524 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000013' has completed.
> > 2011-05-18 16:29:28,589 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000012_0' has finished successfully.
> > 2011-05-18 16:29:28,589 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000012' has completed.
> > 2011-05-18 16:29:28,602 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000001_0' has finished successfully.
> > 2011-05-18 16:29:28,602 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000001' has completed.
> > 2011-05-18 16:29:28,775 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000014_0' has finished successfully.
> > 2011-05-18 16:29:28,775 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000014' has completed.
> > 2011-05-18 16:29:28,909 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000010_0' has finished successfully.
> > 2011-05-18 16:29:28,909 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000010' has completed.
> > 2011-05-18 16:29:28,914 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000007_0' has finished successfully.
> > 2011-05-18 16:29:28,914 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000007' has completed.
> > 2011-05-18 16:29:28,960 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000006_0' has finished successfully.
> > 2011-05-18 16:29:28,960 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000006' has completed.
> > 2011-05-18 16:29:29,148 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000011_0' has finished successfully.
> > 2011-05-18 16:29:29,148 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000011' has completed.
> > 2011-05-18 16:29:29,199 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000008_0' has finished successfully.
> > 2011-05-18 16:29:29,199 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000008' has completed.
> > 2011-05-18 16:29:29,244 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000009_0' has finished successfully.
> > 2011-05-18 16:29:29,244 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000009' has completed.
> > 2011-05-18 16:29:29,274 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000002_0' has finished successfully.
> > 2011-05-18 16:29:29,274 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000002' has completed.
> > 2011-05-18 16:29:29,392 INFO org.apache.hama.bsp.JobInProgress: Taskid
> 'attempt_201105181515_0001_000003_0' has finished successfully.
> > 2011-05-18 16:29:29,392 INFO org.apache.hama.bsp.TaskInProgress: Task
> 'task_201105181515_0001_000003' has completed.
> > 2011-05-18 16:29:29,395 DEBUG org.apache.hama.bsp.JobInProgress: Job
> successfully done.
> > 2011-05-18 16:32:48,365 DEBUG org.apache.hama.bsp.BSPMaster: returns all
> jobs: 1
> > {code}
> >
> >>> Is this still related to the barrier sync?
> >
> > Yes. the problem related with zk node creation/deletion logic in
> enterBarrier() and leaveBarrier() methods. Sometimes they occurs at the same
> time.
> >
> >>> Increasing the timeout won't fix the problem with it.
> >
> > As i mentioned on chat, JVM garbage collection pause causes zk session
> time-out errors.
> >
> >> Development of Shortest Path Finding Algorithm
> >> ----------------------------------------------
> >>
> >>                 Key: HAMA-359
> >>                 URL: https://issues.apache.org/jira/browse/HAMA-359
> >>             Project: Hama
> >>          Issue Type: New Feature
> >>          Components: examples
> >>    Affects Versions: 0.2.0
> >>            Reporter: Edward J. Yoon
> >>            Assignee: Thomas Jungblut
> >>              Labels: gsoc, gsoc2011, mentor
> >>             Fix For: 0.3.0
> >>
> >>         Attachments: HAMA-359-v2.patch, HAMA-359-v3.patch,
> HAMA-359-v4.patch, HAMA-359.patch, eddie.patch
> >>
> >>   Original Estimate: 2016h
> >>  Remaining Estimate: 2016h
> >>
> >> The goal of this project is development of parallel algorithm for
> finding a Shortest Path using Hama BSP.
> >
> > --
> > This message is automatically generated by JIRA.
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
>



-- 
Thomas Jungblut
Berlin

mobile: 0170-3081070

business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message