hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-423) Improve and Refactor Partitioning in the Examples
Date Mon, 22 Aug 2011 02:08:29 GMT

    [ https://issues.apache.org/jira/browse/HAMA-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088510#comment-13088510
] 

Edward J. Yoon commented on HAMA-423:
-------------------------------------

Here is my console results with new patch and textfile on physical 16 nodes cluster. Works
well.

{code}
root@hnode1:/usr/local/src/hama-trunk/core# bin/hama jar ../examples/target/hama-examples-0.4.0-incubating-SNAPSHOT.jar
sssp Umanap edward/sssp-output /user/root/edward/sssp-adjacencylist.txt
Single Source Shortest Path Example:
<Startvertex name> <optional: output path> <optional: path to own adjacency
list textfile!>
Setting default start vertex to "Frankfurt"!
Setting start vertex to Umanap!
Using new output folder: edward/sssp-output
11/08/22 11:00:15 INFO graph.ShortestPaths: Starting data partitioning...
11/08/22 11:01:03 INFO graph.ShortestPaths: Finished!
11/08/22 11:01:04 INFO bsp.BSPJobClient: Running job: job_201108221035_0004
11/08/22 11:01:07 INFO bsp.BSPJobClient: Current supersteps number: 0
11/08/22 11:01:13 INFO bsp.BSPJobClient: Current supersteps number: 2
11/08/22 11:01:16 INFO bsp.BSPJobClient: Current supersteps number: 10
11/08/22 11:01:19 INFO bsp.BSPJobClient: Current supersteps number: 14
11/08/22 11:01:22 INFO bsp.BSPJobClient: Current supersteps number: 18
11/08/22 11:01:28 INFO bsp.BSPJobClient: Current supersteps number: 20
11/08/22 11:01:31 INFO bsp.BSPJobClient: Current supersteps number: 21
11/08/22 11:01:40 INFO bsp.BSPJobClient: Current supersteps number: 23
11/08/22 11:01:43 INFO bsp.BSPJobClient: Current supersteps number: 24
11/08/22 11:01:46 INFO bsp.BSPJobClient: Current supersteps number: 27
11/08/22 11:01:52 INFO bsp.BSPJobClient: Current supersteps number: 30
11/08/22 11:01:58 INFO bsp.BSPJobClient: Current supersteps number: 33
11/08/22 11:02:01 INFO bsp.BSPJobClient: Current supersteps number: 36
11/08/22 11:02:04 INFO bsp.BSPJobClient: Current supersteps number: 39
11/08/22 11:02:07 INFO bsp.BSPJobClient: Current supersteps number: 42
11/08/22 11:02:10 INFO bsp.BSPJobClient: Current supersteps number: 47
11/08/22 11:02:13 INFO bsp.BSPJobClient: Current supersteps number: 50
11/08/22 11:02:16 INFO bsp.BSPJobClient: Current supersteps number: 57
11/08/22 11:02:19 INFO bsp.BSPJobClient: Current supersteps number: 60
11/08/22 11:02:22 INFO bsp.BSPJobClient: Current supersteps number: 68
11/08/22 11:02:25 INFO bsp.BSPJobClient: Current supersteps number: 72
11/08/22 11:02:28 INFO bsp.BSPJobClient: Current supersteps number: 81
11/08/22 11:02:31 INFO bsp.BSPJobClient: Current supersteps number: 85
11/08/22 11:02:34 INFO bsp.BSPJobClient: Current supersteps number: 93
11/08/22 11:02:37 INFO bsp.BSPJobClient: Current supersteps number: 97
11/08/22 11:02:40 INFO bsp.BSPJobClient: Current supersteps number: 102
11/08/22 11:02:43 INFO bsp.BSPJobClient: The total number of supersteps: 102
Job Finished in 99.684 seconds
-------------------- RESULTS --------------------
11/08/22 11:02:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
11/08/22 11:02:43 INFO compress.CodecPool: Got brand-new decompressor
Chan-Santa Cruz | 63422
Samiene | 66036
Pimental | 78866
Chaksom | 84903
Sachiyama | 73654
Itero de la Vega | 67042
....
{code}

BTW, should we print all results?

> Improve and Refactor Partitioning in the Examples
> -------------------------------------------------
>
>                 Key: HAMA-423
>                 URL: https://issues.apache.org/jira/browse/HAMA-423
>             Project: Hama
>          Issue Type: Improvement
>          Components: examples
>    Affects Versions: 0.3.0
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>             Fix For: 0.4.0, 0.5.0
>
>         Attachments: HAMA-423-v1.patch, sickimprovement.PNG
>
>
> Currently partitioning will write a key/value pair for each vertex/adjacent mapping.
> This results in heavy IO writes which actually bloats the file and let the partitioning
take unnecessarily long.
> We should partition directly into the vertex classes and implement a vertex list/array
writable which just writes a single key/value pair for a vertex/all-adjacents mapping.
> In fact we should make it generic, passing a vertex class which should implement the
Writable interface.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message