hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HAMA-129) Improving speed of matrix multiplication
Date Thu, 11 Dec 2008 03:10:44 GMT

    [ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655483#action_12655483
] 

Edward J. Yoon commented on HAMA-129:
-------------------------------------

5000 * 5000 matrix multiplication, blocks 16, 4 node

----
[d8g053:/root/hama-trunk]# bin/hama examples mult fiveA fiveB 16
08/12/11 11:45:22 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix
08/12/11 11:45:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
08/12/11 11:45:22 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/11 11:45:22 WARN mapred.JobClient: No job jar file set.  User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:45:22 INFO mapred.JobClient: Running job: job_200812100956_0045
08/12/11 11:45:23 INFO mapred.JobClient:  map 0% reduce 0%
08/12/11 11:46:05 INFO mapred.JobClient:  map 25% reduce 0%
08/12/11 11:46:14 INFO mapred.JobClient:  map 25% reduce 2%
08/12/11 11:46:15 INFO mapred.JobClient:  map 25% reduce 4%
08/12/11 11:46:16 INFO mapred.JobClient:  map 25% reduce 6%
08/12/11 11:46:18 INFO mapred.JobClient:  map 25% reduce 8%
08/12/11 11:47:58 INFO mapred.JobClient:  map 50% reduce 8%
08/12/11 11:48:02 INFO mapred.JobClient:  map 75% reduce 8%
08/12/11 11:48:03 INFO mapred.JobClient:  map 75% reduce 10%
08/12/11 11:48:07 INFO mapred.JobClient:  map 75% reduce 14%
08/12/11 11:48:08 INFO mapred.JobClient:  map 75% reduce 18%
08/12/11 11:48:11 INFO mapred.JobClient:  map 75% reduce 22%
08/12/11 11:48:15 INFO mapred.JobClient:  map 75% reduce 25%
08/12/11 11:49:11 INFO mapred.JobClient:  map 100% reduce 25%
08/12/11 11:49:17 INFO mapred.JobClient:  map 100% reduce 35%
08/12/11 11:49:18 INFO mapred.JobClient:  map 100% reduce 45%
08/12/11 11:49:20 INFO mapred.JobClient:  map 100% reduce 47%
08/12/11 11:49:21 INFO mapred.JobClient:  map 100% reduce 60%
08/12/11 11:49:22 INFO mapred.JobClient:  map 100% reduce 70%
08/12/11 11:49:25 INFO mapred.JobClient:  map 100% reduce 72%
08/12/11 11:49:26 INFO mapred.JobClient:  map 100% reduce 81%
08/12/11 11:49:27 INFO mapred.JobClient:  map 100% reduce 85%
08/12/11 11:49:30 INFO mapred.JobClient:  map 100% reduce 87%
08/12/11 11:49:31 INFO mapred.JobClient:  map 100% reduce 89%
08/12/11 11:49:33 INFO mapred.JobClient:  map 100% reduce 95%
08/12/11 11:49:36 INFO mapred.JobClient:  map 100% reduce 97%
08/12/11 11:49:39 INFO mapred.JobClient: Job complete: job_200812100956_0045
08/12/11 11:49:39 INFO mapred.JobClient: Counters: 13
08/12/11 11:49:39 INFO mapred.JobClient:   File Systems
08/12/11 11:49:39 INFO mapred.JobClient:     Local bytes read=1656774062
08/12/11 11:49:39 INFO mapred.JobClient:     Local bytes written=2506547810
08/12/11 11:49:39 INFO mapred.JobClient:   Job Counters 
08/12/11 11:49:39 INFO mapred.JobClient:     Launched reduce tasks=4
08/12/11 11:49:39 INFO mapred.JobClient:     Launched map tasks=7
08/12/11 11:49:39 INFO mapred.JobClient:   Map-Reduce Framework
08/12/11 11:49:39 INFO mapred.JobClient:     Reduce input groups=16
08/12/11 11:49:39 INFO mapred.JobClient:     Combine output records=0
08/12/11 11:49:39 INFO mapred.JobClient:     Map input records=5000
08/12/11 11:49:39 INFO mapred.JobClient:     Reduce output records=0
08/12/11 11:49:39 INFO mapred.JobClient:     Map output bytes=844785560
08/12/11 11:49:39 INFO mapred.JobClient:     Map input bytes=0
08/12/11 11:49:39 INFO mapred.JobClient:     Combine input records=0
08/12/11 11:49:39 INFO mapred.JobClient:     Map output records=20000
08/12/11 11:49:39 INFO mapred.JobClient:     Reduce input records=20000
08/12/11 11:49:40 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix
08/12/11 11:49:40 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
08/12/11 11:49:40 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/11 11:49:40 WARN mapred.JobClient: No job jar file set.  User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:49:40 INFO mapred.JobClient: Running job: job_200812100956_0046
08/12/11 11:49:41 INFO mapred.JobClient:  map 0% reduce 0%
08/12/11 11:50:13 INFO mapred.JobClient:  map 25% reduce 0%
08/12/11 11:50:21 INFO mapred.JobClient:  map 25% reduce 2%
08/12/11 11:50:23 INFO mapred.JobClient:  map 25% reduce 4%
08/12/11 11:50:26 INFO mapred.JobClient:  map 25% reduce 6%
08/12/11 11:50:27 INFO mapred.JobClient:  map 25% reduce 8%
08/12/11 11:52:20 INFO mapred.JobClient:  map 50% reduce 8%
08/12/11 11:52:21 INFO mapred.JobClient:  map 75% reduce 8%
08/12/11 11:52:29 INFO mapred.JobClient:  map 75% reduce 12%
08/12/11 11:52:32 INFO mapred.JobClient:  map 75% reduce 16%
08/12/11 11:52:33 INFO mapred.JobClient:  map 75% reduce 25%
08/12/11 11:54:03 INFO mapred.JobClient:  map 100% reduce 25%
08/12/11 11:54:11 INFO mapred.JobClient:  map 100% reduce 35%
08/12/11 11:54:14 INFO mapred.JobClient:  map 100% reduce 37%
08/12/11 11:54:17 INFO mapred.JobClient:  map 100% reduce 47%
08/12/11 11:54:18 INFO mapred.JobClient:  map 100% reduce 72%
08/12/11 11:54:20 INFO mapred.JobClient:  map 100% reduce 75%
08/12/11 11:54:22 INFO mapred.JobClient:  map 100% reduce 77%
08/12/11 11:54:23 INFO mapred.JobClient:  map 100% reduce 83%
08/12/11 11:54:24 INFO mapred.JobClient:  map 100% reduce 85%
08/12/11 11:54:25 INFO mapred.JobClient:  map 100% reduce 87%
08/12/11 11:54:26 INFO mapred.JobClient:  map 100% reduce 89%
08/12/11 11:54:27 INFO mapred.JobClient:  map 100% reduce 95%
08/12/11 11:54:32 INFO mapred.JobClient: Job complete: job_200812100956_0046
08/12/11 11:54:32 INFO mapred.JobClient: Counters: 13
08/12/11 11:54:32 INFO mapred.JobClient:   File Systems
08/12/11 11:54:32 INFO mapred.JobClient:     Local bytes read=1656774062
08/12/11 11:54:32 INFO mapred.JobClient:     Local bytes written=2506547810
08/12/11 11:54:32 INFO mapred.JobClient:   Job Counters 
08/12/11 11:54:32 INFO mapred.JobClient:     Launched reduce tasks=5
08/12/11 11:54:32 INFO mapred.JobClient:     Launched map tasks=7
08/12/11 11:54:32 INFO mapred.JobClient:   Map-Reduce Framework
08/12/11 11:54:32 INFO mapred.JobClient:     Reduce input groups=16
08/12/11 11:54:32 INFO mapred.JobClient:     Combine output records=0
08/12/11 11:54:32 INFO mapred.JobClient:     Map input records=5000
08/12/11 11:54:32 INFO mapred.JobClient:     Reduce output records=0
08/12/11 11:54:32 INFO mapred.JobClient:     Map output bytes=844785560
08/12/11 11:54:32 INFO mapred.JobClient:     Map input bytes=0
08/12/11 11:54:32 INFO mapred.JobClient:     Combine input records=0
08/12/11 11:54:32 INFO mapred.JobClient:     Map output records=20000
08/12/11 11:54:32 INFO mapred.JobClient:     Reduce input records=20000
08/12/11 11:54:32 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/11 11:54:38 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randbcfpq
08/12/11 11:54:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
08/12/11 11:54:38 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/11 11:54:38 WARN mapred.JobClient: No job jar file set.  User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:54:38 INFO mapred.JobClient: Running job: job_200812100956_0047
08/12/11 11:54:39 INFO mapred.JobClient:  map 0% reduce 0%
08/12/11 11:57:30 INFO mapred.JobClient:  map 50% reduce 0%
08/12/11 11:57:37 INFO mapred.JobClient:  map 50% reduce 4%
08/12/11 11:57:41 INFO mapred.JobClient:  map 50% reduce 8%
08/12/11 11:57:42 INFO mapred.JobClient:  map 50% reduce 12%
08/12/11 11:57:47 INFO mapred.JobClient:  map 50% reduce 16%
08/12/11 12:00:31 INFO mapred.JobClient:  map 100% reduce 16%
08/12/11 12:00:38 INFO mapred.JobClient:  map 100% reduce 29%
08/12/11 12:00:43 INFO mapred.JobClient:  map 100% reduce 42%
08/12/11 12:00:46 INFO mapred.JobClient:  map 100% reduce 46%
08/12/11 12:00:47 INFO mapred.JobClient:  map 100% reduce 59%
08/12/11 12:00:51 INFO mapred.JobClient:  map 100% reduce 68%
08/12/11 12:07:08 INFO mapred.JobClient:  map 100% reduce 76%
08/12/11 12:07:39 INFO mapred.JobClient:  map 100% reduce 84%
08/12/11 12:07:40 INFO mapred.JobClient:  map 100% reduce 92%
08/12/11 12:07:45 INFO mapred.JobClient: Job complete: job_200812100956_0047
08/12/11 12:07:45 INFO mapred.JobClient: Counters: 13
08/12/11 12:07:45 INFO mapred.JobClient:   File Systems
08/12/11 12:07:45 INFO mapred.JobClient:     Local bytes read=1639585414
08/12/11 12:07:45 INFO mapred.JobClient:     Local bytes written=2421198052
08/12/11 12:07:45 INFO mapred.JobClient:   Job Counters 
08/12/11 12:07:45 INFO mapred.JobClient:     Launched reduce tasks=5
08/12/11 12:07:45 INFO mapred.JobClient:     Launched map tasks=3
08/12/11 12:07:45 INFO mapred.JobClient:   Map-Reduce Framework
08/12/11 12:07:45 INFO mapred.JobClient:     Reduce input groups=4
08/12/11 12:07:45 INFO mapred.JobClient:     Combine output records=0
08/12/11 12:07:45 INFO mapred.JobClient:     Map input records=4
08/12/11 12:07:45 INFO mapred.JobClient:     Reduce output records=20000
08/12/11 12:07:45 INFO mapred.JobClient:     Map output bytes=800808832
08/12/11 12:07:45 INFO mapred.JobClient:     Map input bytes=0
08/12/11 12:07:45 INFO mapred.JobClient:     Combine input records=0
08/12/11 12:07:45 INFO mapred.JobClient:     Map output records=64
08/12/11 12:07:45 INFO mapred.JobClient:     Reduce input records=64


> Improving speed of matrix multiplication
> ----------------------------------------
>
>                 Key: HAMA-129
>                 URL: https://issues.apache.org/jira/browse/HAMA-129
>             Project: Hama
>          Issue Type: Improvement
>          Components: implementation
>    Affects Versions: 0.1.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.1.0
>
>         Attachments: HAMA-129.patch, HAMA-129_v01.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError:
Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message