incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Block multiplication.
Date Mon, 17 Nov 2008 05:35:54 GMT
Yes, I think so.

We should find the best way to create blocked dense matrix.

On Mon, Nov 17, 2008 at 2:24 PM, Samuel Guo <guosijie@gmail.com> wrote:
> hmm, It sounds interesting.
>
> Will the blocking operation be a bottle-neck when the matrix becomes large ?
>
> On Mon, Nov 17, 2008 at 12:08 PM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>
>> {{{
>>  // Generate matrix with random elements
>>  DenseMatrix a = DenseMatrix.random(conf, 1000, 1000);
>>  DenseMatrix b = DenseMatrix.random(conf, 1000, 1000);
>>
>> ---------------------
>>
>>  // The type of the Matrix to be blocked, must be dense.
>>  a.blocking();
>>  b.blocking();
>>
>>  DenseMatrix c = a.mult(b);
>> }}}
>>
>> So, I designed as above.
>> If we want to save a matrix A to table and it'll be continuosly
>> updated by something, Blocked dense matrix seems not fit. And, we have
>> a blocking job once before operate something.
>>
>> On Mon, Nov 17, 2008 at 12:59 PM, Edward J. Yoon <edwardyoon@apache.org>
>> wrote:
>> > I think blocked dense matrix have a character of read-only. If we
>> > additionally put some elements, many blocks should be re-arranged.
>> >
>> > On Mon, Nov 17, 2008 at 12:27 PM, Samuel Guo <guosijie@gmail.com> wrote:
>> >> Why not we construct the matrix using block format?
>> >> It's a strange operation to block a DenseMatrix to a block matrix
>> storage
>> >> format.
>> >>
>> >> BTW, blocking operation seems a expensive operation in Client.
>> >>
>> >> On Mon, Nov 17, 2008 at 10:48 AM, Edward J. Yoon <edwardyoon@apache.org
>> >wrote:
>> >>
>> >>> When I tried to multiply the 1000 * 1000, I received below massage
>> >>> before start the mapreduce.
>> >>>
>> >>> This is a issue of blocking job. It'll be handled on HAMA-107.
>> >>>
>> >>> 08/11/17 11:08:09 INFO hama.AbstractMatrix: Initializing the matrix
>> >>> storage.
>> >>> 08/11/17 11:08:16 INFO hama.AbstractMatrix: Create Matrix
>> >>> DenseMatrix_randhzngs
>> >>> 08/11/17 11:08:16 INFO hama.AbstractMatrix: Create the 1000 * 1000
>> >>> random matrix : DenseMatrix_randhzngs
>> >>> java.lang.RuntimeException:
>> >>> org.apache.hadoop.hbase.UnknownScannerException:
>> >>> org.apache.hadoop.hbase.UnknownScannerException: Name:
>> >>> 1659554672448075830
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077)
>> >>>        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>> >>>        at
>> >>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>        at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>        at
>> >>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> >>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> >>>
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1195)
>> >>>        at org.apache.hama.DenseMatrix.blocking(DenseMatrix.java:390)
>> >>>        at
>> >>>
>> org.apache.hama.examples.MatrixMultiplication.main(MatrixMultiplication.java:46)
>> >>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>        at
>> >>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>        at
>> >>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>        at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>        at
>> >>>
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> >>>        at
>> >>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
>> >>>        at
>> >>> org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:30)
>> >>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>        at
>> >>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>        at
>> >>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>        at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>> >>>        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>> >>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >>>        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>> >>> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
>> >>> org.apache.hadoop.hbase.UnknownScannerException: Name:
>> >>> 1659554672448075830
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077)
>> >>>        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>> >>>        at
>> >>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>        at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>        at
>> >>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> >>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> >>>
>> >>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> >>> Method)
>> >>>        at
>> >>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >>>        at
>> >>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >>>        at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856)
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155)
>> >>>        at
>> >>>
>> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1192)
>> >>>        ... 18 more
>> >>>
>> >>>
>> >>> On Fri, Nov 14, 2008 at 6:53 PM, Edward J. Yoon <edwardyoon@apache.org
>> >
>> >>> wrote:
>> >>> > mat-mat mult operation time was dramatically reduced. (to 12 seconds
>> >>> > from 300 seconds on same condition) but code is not good (because
by
>> >>> > me). please review this.
>> >>> >
>> >>> > And, we'll start the research about block decomposition.
>> >>> > --
>> >>> > Best Regards, Edward J. Yoon @ NHN, corp.
>> >>> > edwardyoon@apache.org
>> >>> > http://blog.udanax.org
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon @ NHN, corp.
>> >>> edwardyoon@apache.org
>> >>> http://blog.udanax.org
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon @ NHN, corp.
>> > edwardyoon@apache.org
>> > http://blog.udanax.org
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message