Return-Path: Delivered-To: apmail-incubator-hama-dev-archive@locus.apache.org Received: (qmail 71352 invoked from network); 11 Dec 2008 03:11:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 11 Dec 2008 03:11:05 -0000 Received: (qmail 5945 invoked by uid 500); 11 Dec 2008 03:11:17 -0000 Delivered-To: apmail-incubator-hama-dev-archive@incubator.apache.org Received: (qmail 5925 invoked by uid 500); 11 Dec 2008 03:11:17 -0000 Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-dev@incubator.apache.org Received: (qmail 5914 invoked by uid 99); 11 Dec 2008 03:11:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Dec 2008 19:11:17 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Dec 2008 03:11:04 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 51186234C35F for ; Wed, 10 Dec 2008 19:10:44 -0800 (PST) Message-ID: <455364059.1228965044331.JavaMail.jira@brutus> Date: Wed, 10 Dec 2008 19:10:44 -0800 (PST) From: "Edward J. Yoon (JIRA)" To: hama-dev@incubator.apache.org Subject: [jira] Commented: (HAMA-129) Improving speed of matrix multiplication In-Reply-To: <1276950529.1228808504272.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655483#action_12655483 ] Edward J. Yoon commented on HAMA-129: ------------------------------------- 5000 * 5000 matrix multiplication, blocks 16, 4 node ---- [d8g053:/root/hama-trunk]# bin/hama examples mult fiveA fiveB 16 08/12/11 11:45:22 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix 08/12/11 11:45:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 08/12/11 11:45:22 WARN mapred.JobClient: Use genericOptions for the option -libjars 08/12/11 11:45:22 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 08/12/11 11:45:22 INFO mapred.JobClient: Running job: job_200812100956_0045 08/12/11 11:45:23 INFO mapred.JobClient: map 0% reduce 0% 08/12/11 11:46:05 INFO mapred.JobClient: map 25% reduce 0% 08/12/11 11:46:14 INFO mapred.JobClient: map 25% reduce 2% 08/12/11 11:46:15 INFO mapred.JobClient: map 25% reduce 4% 08/12/11 11:46:16 INFO mapred.JobClient: map 25% reduce 6% 08/12/11 11:46:18 INFO mapred.JobClient: map 25% reduce 8% 08/12/11 11:47:58 INFO mapred.JobClient: map 50% reduce 8% 08/12/11 11:48:02 INFO mapred.JobClient: map 75% reduce 8% 08/12/11 11:48:03 INFO mapred.JobClient: map 75% reduce 10% 08/12/11 11:48:07 INFO mapred.JobClient: map 75% reduce 14% 08/12/11 11:48:08 INFO mapred.JobClient: map 75% reduce 18% 08/12/11 11:48:11 INFO mapred.JobClient: map 75% reduce 22% 08/12/11 11:48:15 INFO mapred.JobClient: map 75% reduce 25% 08/12/11 11:49:11 INFO mapred.JobClient: map 100% reduce 25% 08/12/11 11:49:17 INFO mapred.JobClient: map 100% reduce 35% 08/12/11 11:49:18 INFO mapred.JobClient: map 100% reduce 45% 08/12/11 11:49:20 INFO mapred.JobClient: map 100% reduce 47% 08/12/11 11:49:21 INFO mapred.JobClient: map 100% reduce 60% 08/12/11 11:49:22 INFO mapred.JobClient: map 100% reduce 70% 08/12/11 11:49:25 INFO mapred.JobClient: map 100% reduce 72% 08/12/11 11:49:26 INFO mapred.JobClient: map 100% reduce 81% 08/12/11 11:49:27 INFO mapred.JobClient: map 100% reduce 85% 08/12/11 11:49:30 INFO mapred.JobClient: map 100% reduce 87% 08/12/11 11:49:31 INFO mapred.JobClient: map 100% reduce 89% 08/12/11 11:49:33 INFO mapred.JobClient: map 100% reduce 95% 08/12/11 11:49:36 INFO mapred.JobClient: map 100% reduce 97% 08/12/11 11:49:39 INFO mapred.JobClient: Job complete: job_200812100956_0045 08/12/11 11:49:39 INFO mapred.JobClient: Counters: 13 08/12/11 11:49:39 INFO mapred.JobClient: File Systems 08/12/11 11:49:39 INFO mapred.JobClient: Local bytes read=1656774062 08/12/11 11:49:39 INFO mapred.JobClient: Local bytes written=2506547810 08/12/11 11:49:39 INFO mapred.JobClient: Job Counters 08/12/11 11:49:39 INFO mapred.JobClient: Launched reduce tasks=4 08/12/11 11:49:39 INFO mapred.JobClient: Launched map tasks=7 08/12/11 11:49:39 INFO mapred.JobClient: Map-Reduce Framework 08/12/11 11:49:39 INFO mapred.JobClient: Reduce input groups=16 08/12/11 11:49:39 INFO mapred.JobClient: Combine output records=0 08/12/11 11:49:39 INFO mapred.JobClient: Map input records=5000 08/12/11 11:49:39 INFO mapred.JobClient: Reduce output records=0 08/12/11 11:49:39 INFO mapred.JobClient: Map output bytes=844785560 08/12/11 11:49:39 INFO mapred.JobClient: Map input bytes=0 08/12/11 11:49:39 INFO mapred.JobClient: Combine input records=0 08/12/11 11:49:39 INFO mapred.JobClient: Map output records=20000 08/12/11 11:49:39 INFO mapred.JobClient: Reduce input records=20000 08/12/11 11:49:40 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix 08/12/11 11:49:40 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 08/12/11 11:49:40 WARN mapred.JobClient: Use genericOptions for the option -libjars 08/12/11 11:49:40 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 08/12/11 11:49:40 INFO mapred.JobClient: Running job: job_200812100956_0046 08/12/11 11:49:41 INFO mapred.JobClient: map 0% reduce 0% 08/12/11 11:50:13 INFO mapred.JobClient: map 25% reduce 0% 08/12/11 11:50:21 INFO mapred.JobClient: map 25% reduce 2% 08/12/11 11:50:23 INFO mapred.JobClient: map 25% reduce 4% 08/12/11 11:50:26 INFO mapred.JobClient: map 25% reduce 6% 08/12/11 11:50:27 INFO mapred.JobClient: map 25% reduce 8% 08/12/11 11:52:20 INFO mapred.JobClient: map 50% reduce 8% 08/12/11 11:52:21 INFO mapred.JobClient: map 75% reduce 8% 08/12/11 11:52:29 INFO mapred.JobClient: map 75% reduce 12% 08/12/11 11:52:32 INFO mapred.JobClient: map 75% reduce 16% 08/12/11 11:52:33 INFO mapred.JobClient: map 75% reduce 25% 08/12/11 11:54:03 INFO mapred.JobClient: map 100% reduce 25% 08/12/11 11:54:11 INFO mapred.JobClient: map 100% reduce 35% 08/12/11 11:54:14 INFO mapred.JobClient: map 100% reduce 37% 08/12/11 11:54:17 INFO mapred.JobClient: map 100% reduce 47% 08/12/11 11:54:18 INFO mapred.JobClient: map 100% reduce 72% 08/12/11 11:54:20 INFO mapred.JobClient: map 100% reduce 75% 08/12/11 11:54:22 INFO mapred.JobClient: map 100% reduce 77% 08/12/11 11:54:23 INFO mapred.JobClient: map 100% reduce 83% 08/12/11 11:54:24 INFO mapred.JobClient: map 100% reduce 85% 08/12/11 11:54:25 INFO mapred.JobClient: map 100% reduce 87% 08/12/11 11:54:26 INFO mapred.JobClient: map 100% reduce 89% 08/12/11 11:54:27 INFO mapred.JobClient: map 100% reduce 95% 08/12/11 11:54:32 INFO mapred.JobClient: Job complete: job_200812100956_0046 08/12/11 11:54:32 INFO mapred.JobClient: Counters: 13 08/12/11 11:54:32 INFO mapred.JobClient: File Systems 08/12/11 11:54:32 INFO mapred.JobClient: Local bytes read=1656774062 08/12/11 11:54:32 INFO mapred.JobClient: Local bytes written=2506547810 08/12/11 11:54:32 INFO mapred.JobClient: Job Counters 08/12/11 11:54:32 INFO mapred.JobClient: Launched reduce tasks=5 08/12/11 11:54:32 INFO mapred.JobClient: Launched map tasks=7 08/12/11 11:54:32 INFO mapred.JobClient: Map-Reduce Framework 08/12/11 11:54:32 INFO mapred.JobClient: Reduce input groups=16 08/12/11 11:54:32 INFO mapred.JobClient: Combine output records=0 08/12/11 11:54:32 INFO mapred.JobClient: Map input records=5000 08/12/11 11:54:32 INFO mapred.JobClient: Reduce output records=0 08/12/11 11:54:32 INFO mapred.JobClient: Map output bytes=844785560 08/12/11 11:54:32 INFO mapred.JobClient: Map input bytes=0 08/12/11 11:54:32 INFO mapred.JobClient: Combine input records=0 08/12/11 11:54:32 INFO mapred.JobClient: Map output records=20000 08/12/11 11:54:32 INFO mapred.JobClient: Reduce input records=20000 08/12/11 11:54:32 INFO hama.AbstractMatrix: Initializing the matrix storage. 08/12/11 11:54:38 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randbcfpq 08/12/11 11:54:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 08/12/11 11:54:38 WARN mapred.JobClient: Use genericOptions for the option -libjars 08/12/11 11:54:38 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 08/12/11 11:54:38 INFO mapred.JobClient: Running job: job_200812100956_0047 08/12/11 11:54:39 INFO mapred.JobClient: map 0% reduce 0% 08/12/11 11:57:30 INFO mapred.JobClient: map 50% reduce 0% 08/12/11 11:57:37 INFO mapred.JobClient: map 50% reduce 4% 08/12/11 11:57:41 INFO mapred.JobClient: map 50% reduce 8% 08/12/11 11:57:42 INFO mapred.JobClient: map 50% reduce 12% 08/12/11 11:57:47 INFO mapred.JobClient: map 50% reduce 16% 08/12/11 12:00:31 INFO mapred.JobClient: map 100% reduce 16% 08/12/11 12:00:38 INFO mapred.JobClient: map 100% reduce 29% 08/12/11 12:00:43 INFO mapred.JobClient: map 100% reduce 42% 08/12/11 12:00:46 INFO mapred.JobClient: map 100% reduce 46% 08/12/11 12:00:47 INFO mapred.JobClient: map 100% reduce 59% 08/12/11 12:00:51 INFO mapred.JobClient: map 100% reduce 68% 08/12/11 12:07:08 INFO mapred.JobClient: map 100% reduce 76% 08/12/11 12:07:39 INFO mapred.JobClient: map 100% reduce 84% 08/12/11 12:07:40 INFO mapred.JobClient: map 100% reduce 92% 08/12/11 12:07:45 INFO mapred.JobClient: Job complete: job_200812100956_0047 08/12/11 12:07:45 INFO mapred.JobClient: Counters: 13 08/12/11 12:07:45 INFO mapred.JobClient: File Systems 08/12/11 12:07:45 INFO mapred.JobClient: Local bytes read=1639585414 08/12/11 12:07:45 INFO mapred.JobClient: Local bytes written=2421198052 08/12/11 12:07:45 INFO mapred.JobClient: Job Counters 08/12/11 12:07:45 INFO mapred.JobClient: Launched reduce tasks=5 08/12/11 12:07:45 INFO mapred.JobClient: Launched map tasks=3 08/12/11 12:07:45 INFO mapred.JobClient: Map-Reduce Framework 08/12/11 12:07:45 INFO mapred.JobClient: Reduce input groups=4 08/12/11 12:07:45 INFO mapred.JobClient: Combine output records=0 08/12/11 12:07:45 INFO mapred.JobClient: Map input records=4 08/12/11 12:07:45 INFO mapred.JobClient: Reduce output records=20000 08/12/11 12:07:45 INFO mapred.JobClient: Map output bytes=800808832 08/12/11 12:07:45 INFO mapred.JobClient: Map input bytes=0 08/12/11 12:07:45 INFO mapred.JobClient: Combine input records=0 08/12/11 12:07:45 INFO mapred.JobClient: Map output records=64 08/12/11 12:07:45 INFO mapred.JobClient: Reduce input records=64 > Improving speed of matrix multiplication > ---------------------------------------- > > Key: HAMA-129 > URL: https://issues.apache.org/jira/browse/HAMA-129 > Project: Hama > Issue Type: Improvement > Components: implementation > Affects Versions: 0.1.0 > Reporter: Edward J. Yoon > Assignee: Edward J. Yoon > Fix For: 0.1.0 > > Attachments: HAMA-129.patch, HAMA-129_v01.patch > > > We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big. > I'll write the formulas and partition them as PIEstimator. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.