hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HAMA-932) Use Kryo Serializer for message and bundle serializations.
Date Tue, 10 Mar 2015 09:34:38 GMT

     [ https://issues.apache.org/jira/browse/HAMA-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Edward J. Yoon updated HAMA-932:
--------------------------------
    Attachment: kryo.patch

> Use Kryo Serializer for message and bundle serializations.
> ----------------------------------------------------------
>
>                 Key: HAMA-932
>                 URL: https://issues.apache.org/jira/browse/HAMA-932
>             Project: Hama
>          Issue Type: Improvement
>          Components: bsp core
>    Affects Versions: 0.6.4
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.7.0
>
>         Attachments: kryo.patch
>
>
> As we already know, kryo serializer shows quite awesome performance. I suggest that we
use kryo serializer for everything.
> Attached patch uses kryo for message serialization and below is its results:
> {code}
> 15/03/10 18:08:08 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
> 15/03/10 18:08:08 INFO bsp.FileInputFormat: Total input paths to process : 3
> 15/03/10 18:08:09 INFO bsp.BSPJobClient: Running job: job_201503101804_0007
> 15/03/10 18:08:12 INFO bsp.BSPJobClient: Current supersteps number: 0
> 15/03/10 18:09:34 INFO bsp.BSPJobClient: Current supersteps number: 1
> 15/03/10 18:11:24 INFO bsp.BSPJobClient: Current supersteps number: 2
> 15/03/10 18:11:27 INFO bsp.BSPJobClient: Current supersteps number: 3
> 15/03/10 18:14:33 INFO bsp.BSPJobClient: Current supersteps number: 4
> 15/03/10 18:14:36 INFO bsp.BSPJobClient: Current supersteps number: 5
> 15/03/10 18:17:20 INFO bsp.BSPJobClient: Current supersteps number: 7
> 15/03/10 18:18:12 INFO bsp.BSPJobClient: Current supersteps number: 9
> 15/03/10 18:18:15 INFO bsp.BSPJobClient: Current supersteps number: 11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: The total number of supersteps: 11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: Counters: 11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:   org.apache.hama.graph.GraphJobRunner$GraphJobCounter
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     ITERATIONS=4
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     INPUT_VERTICES=30000
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:   org.apache.hama.bsp.JobInProgress$JobCounter
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     SUPERSTEPS=11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=3
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:   org.apache.hama.bsp.BSPPeerImpl$PeerCounter
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=33
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=596707
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     IO_BYTES_READ=84870046
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_SENT=44717766
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     TOTAL_MESSAGE_BYTES_TRANSFERED=361694719
> 15/03/10 18:18:30 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_RECEIVED=270035
> Job Finished in 639.248 seconds
> .... 
> Kryo version:
> 15/03/10 18:23:17 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
> 15/03/10 18:23:17 INFO bsp.FileInputFormat: Total input paths to process : 3
> 15/03/10 18:23:17 INFO bsp.BSPJobClient: Running job: job_201503101822_0001
> 15/03/10 18:23:20 INFO bsp.BSPJobClient: Current supersteps number: 0
> 15/03/10 18:23:35 INFO bsp.BSPJobClient: Current supersteps number: 1
> 15/03/10 18:24:16 INFO bsp.BSPJobClient: Current supersteps number: 2
> 15/03/10 18:24:19 INFO bsp.BSPJobClient: Current supersteps number: 3
> 15/03/10 18:25:23 INFO bsp.BSPJobClient: Current supersteps number: 4
> 15/03/10 18:25:26 INFO bsp.BSPJobClient: Current supersteps number: 5
> 15/03/10 18:26:47 INFO bsp.BSPJobClient: Current supersteps number: 7
> 15/03/10 18:26:53 INFO bsp.BSPJobClient: Current supersteps number: 9
> 15/03/10 18:26:56 INFO bsp.BSPJobClient: Current supersteps number: 11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: The total number of supersteps: 11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: Counters: 11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:   org.apache.hama.graph.GraphJobRunner$GraphJobCounter
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     ITERATIONS=4
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     INPUT_VERTICES=30000
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:   org.apache.hama.bsp.JobInProgress$JobCounter
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     SUPERSTEPS=11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=3
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:   org.apache.hama.bsp.BSPPeerImpl$PeerCounter
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=33
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=185464
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     IO_BYTES_READ=84870046
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_SENT=44717766
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     TOTAL_MESSAGE_BYTES_TRANSFERED=573865062
> 15/03/10 18:27:10 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_RECEIVED=270039
> Job Finished in 250.217 seconds
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message