incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HAMA-129) Improving speed of matrix multiplication
Date Wed, 14 Jan 2009 00:14:59 GMT

    [ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663565#action_12663565
] 

Edward J. Yoon commented on HAMA-129:
-------------------------------------

More large:

----
09/01/13 20:49:23 INFO mapred.JobClient:  map 100% reduce 40%
09/01/13 20:49:23 INFO mapred.JobClient: Task Id : attempt_200901131739_0005_r_000017_2, Status
: FAILED
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
61.247.201.165:60020 for region DenseMatrix_randwkzjh,,1231846579535, row '000000000000000',
but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

java.io.IOException: java.io.IOException: Could not read from stream
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
        at org.apache.hadoop.io.Text.readString(Text.java:400)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)


        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:843)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1372)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1316)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1296)
        at org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71)
        at org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51)
        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405)
        at org.apache.hama.algebra.BlockCyclicMultiplyReduce.reduce(BlockCyclicMultiplyReduce.java:83)
        at org.apache.hama.algebra.BlockCyclicMultiplyReduce.reduce(BlockCyclicMultiplyReduce.java:39)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
        at org.apache.hadoop.mapred.Child.main(Child.java:155)

09/01/13 20:49:29 INFO mapred.JobClient:  map 100% reduce 38%
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217)
        at org.apache.hama.util.JobManager.execute(JobManager.java:33)
        at org.apache.hama.DenseMatrix.mult(DenseMatrix.java:421)
        at org.apache.hama.examples.MatrixMultiplication.main(MatrixMultiplication.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
        at org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:32)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)


> Improving speed of matrix multiplication
> ----------------------------------------
>
>                 Key: HAMA-129
>                 URL: https://issues.apache.org/jira/browse/HAMA-129
>             Project: Hama
>          Issue Type: Improvement
>          Components: implementation
>    Affects Versions: 0.1.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.1.0
>
>         Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch,
HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch,
patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError:
Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message