incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Block multiplication.
Date Mon, 17 Nov 2008 04:08:30 GMT
{{{
  // Generate matrix with random elements
  DenseMatrix a = DenseMatrix.random(conf, 1000, 1000);
  DenseMatrix b = DenseMatrix.random(conf, 1000, 1000);

---------------------

  // The type of the Matrix to be blocked, must be dense.
  a.blocking();
  b.blocking();

  DenseMatrix c = a.mult(b);
}}}

So, I designed as above.
If we want to save a matrix A to table and it'll be continuosly
updated by something, Blocked dense matrix seems not fit. And, we have
a blocking job once before operate something.

On Mon, Nov 17, 2008 at 12:59 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> I think blocked dense matrix have a character of read-only. If we
> additionally put some elements, many blocks should be re-arranged.
>
> On Mon, Nov 17, 2008 at 12:27 PM, Samuel Guo <guosijie@gmail.com> wrote:
>> Why not we construct the matrix using block format?
>> It's a strange operation to block a DenseMatrix to a block matrix storage
>> format.
>>
>> BTW, blocking operation seems a expensive operation in Client.
>>
>> On Mon, Nov 17, 2008 at 10:48 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>>
>>> When I tried to multiply the 1000 * 1000, I received below massage
>>> before start the mapreduce.
>>>
>>> This is a issue of blocking job. It'll be handled on HAMA-107.
>>>
>>> 08/11/17 11:08:09 INFO hama.AbstractMatrix: Initializing the matrix
>>> storage.
>>> 08/11/17 11:08:16 INFO hama.AbstractMatrix: Create Matrix
>>> DenseMatrix_randhzngs
>>> 08/11/17 11:08:16 INFO hama.AbstractMatrix: Create the 1000 * 1000
>>> random matrix : DenseMatrix_randhzngs
>>> java.lang.RuntimeException:
>>> org.apache.hadoop.hbase.UnknownScannerException:
>>> org.apache.hadoop.hbase.UnknownScannerException: Name:
>>> 1659554672448075830
>>>        at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077)
>>>        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>>>        at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at
>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>>>
>>>        at
>>> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1195)
>>>        at org.apache.hama.DenseMatrix.blocking(DenseMatrix.java:390)
>>>        at
>>> org.apache.hama.examples.MatrixMultiplication.main(MatrixMultiplication.java:46)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at
>>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>>>        at
>>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
>>>        at
>>> org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:30)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>>>        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>>> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
>>> org.apache.hadoop.hbase.UnknownScannerException: Name:
>>> 1659554672448075830
>>>        at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077)
>>>        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>>>        at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at
>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>>>
>>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>        at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>        at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>        at
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
>>>        at
>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856)
>>>        at
>>> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155)
>>>        at
>>> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1192)
>>>        ... 18 more
>>>
>>>
>>> On Fri, Nov 14, 2008 at 6:53 PM, Edward J. Yoon <edwardyoon@apache.org>
>>> wrote:
>>> > mat-mat mult operation time was dramatically reduced. (to 12 seconds
>>> > from 300 seconds on same condition) but code is not good (because by
>>> > me). please review this.
>>> >
>>> > And, we'll start the research about block decomposition.
>>> > --
>>> > Best Regards, Edward J. Yoon @ NHN, corp.
>>> > edwardyoon@apache.org
>>> > http://blog.udanax.org
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon @ NHN, corp.
>>> edwardyoon@apache.org
>>> http://blog.udanax.org
>>>
>>
>
>
>
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> edwardyoon@apache.org
> http://blog.udanax.org
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message