hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Hama "add, multi, norms, similarity" examples problems
Date Tue, 16 Feb 2010 15:25:13 GMT
Sorry, I just noticed that the addition of sparse matrices is not
implemented yet. I'll fix it soon.

Currently, an addition of dense matrices will only be run.

# bin/hama examples rand 100 100 100% testA
# bin/hama examples rand 100 100 100% testB
# bin/hama examples add testA testB

On Tue, Feb 16, 2010 at 11:07 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Hi, Andrew
>
>>> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
>
> Eh, It seems an un-expected exception. Let me check it out.
>
>>> Another question is about lack of "breadth first search" sample,
>>> is this a sub-project of next release?
>
> Using M/R iterations, you can simply implement it --
> http://blog.udanax.org/2009/02/breadth-first-search-mapreduce.html
>
> But, an M/R iterative job is *awfully* slow, so I think it's not worth
> to implement.
>
> Instead, we trying to implement a BSP-based graph computing framework
> to improve the performance of graph algorithms on Hadoop.
>
> On Tue, Feb 16, 2010 at 10:41 PM, Andrew Milkowski
> <andrewmilkowski@gmail.com> wrote:
>> Hi there
>>
>> Have following working setup:
>>
>> hadoop 0.20.1
>> hbase 0.20.3
>> hama latest SVN trunk version
>>
>> Notes:
>>
>> had to replace and rebuild hama with new hbase lib (upgraded to version
>> 0.20.3)
>>
>> 1. successfully rand hama example to generate random matrix
>>
>> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA
>>
>> Note: verified hbase table was generated (via shell list and scan)
>>
>> 2. moved on to other examples, generated 2 random matrices:
>>
>> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA
>> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixB
>>
>> then an attempt at distributed addition:
>>
>> $HAMA_HOME/bin/hama examples add1  matrixA matrixB
>>
>>
>> 3. in similar venue tried multi, norms and and similarity (all generated
>> exceptions)
>>
>> Exceptions appear to be hama code related (and not environmentally specific)
>>
>> Culprit could be in improper usage of input matrix data to add,multi, norms
>> and similarity.
>>
>> Another question is about lack of "breadth first search" sample, is this a
>> sub-project of next release?
>>
>> thanks
>>
>> below attached console stack trace of bash-3.2$ ./run_examples.sh (Note:
>> ignore  INFO zookeeper.ClientCnxn: Exception while closing send thread for
>> session... exception that is a known "INFO" level exception present in
>> zookeeper client/server socket code fragment)
>>
>>
>> --------------------------------------------
>>
>> run_examples.sh:
>>
>> #!/bin/bash
>>
>> export HAMA_JAR="${HAMA_HOME}/hama-0.2.0-dev-examples.jar"
>>
>> # cygwin path translation
>> #if $cygwin; then
>> #  HAMA_CLASSPATH=`cygpath -p -w "$HAMA_CLASSPATH"`
>> #fi
>>
>> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA
>>
>> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixB
>>
>> $HAMA_HOME/bin/hama examples add matrixA matrixB
>>
>> exit $?
>>
>>
>>
>>
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:host.name
>> =192.168.0.11
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.version=1.6.0_17
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.vendor=Apple Inc.
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.compiler=<NA>
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
>> OS X
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:os.arch=x86_64
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:os.version=10.6.2
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:user.name
>> =hadoop
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:user.home=/Users/hadoop
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
>> environment:user.dir=/opt/local/hadoop/jobs/hama
>> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Initiating client connection,
>> connectString=localhost:2181 sessionTimeout=60000
>> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@2efb56b1
>> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
>> is false
>> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Attempting connection to server
>> localhost/127.0.0.1:2181
>> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Priming connection to
>> java.nio.channels.SocketChannel[connected
>> local=/127.0.0.1:58238remote=localhost/
>> 127.0.0.1:2181]
>> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Server connection successful
>> 10/02/16 08:28:30 INFO matrix.AbstractMatrix: Initializing the matrix
>> storage.
>> 10/02/16 08:28:32 INFO matrix.AbstractMatrix: Create Matrix
>> SparseMatrix_randliqns
>> 10/02/16 08:28:32 INFO matrix.AbstractMatrix: Create the 100 * 100 random
>> matrix : SparseMatrix_randliqns
>> Wrote input for Map #0
>> Wrote input for Map #1
>> Wrote input for Map #2
>> Wrote input for Map #3
>> Wrote input for Map #4
>> Wrote input for Map #5
>> Wrote input for Map #6
>> Wrote input for Map #7
>> Wrote input for Map #8
>> Wrote input for Map #9
>> 10/02/16 08:28:33 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 10/02/16 08:28:33 WARN mapred.JobClient: No job jar file set.  User classes
>> may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 10/02/16 08:28:33 INFO input.FileInputFormat: Total input paths to process :
>> 10
>> 10/02/16 08:28:34 INFO mapred.JobClient: Running job: job_201002160823_0001
>> 10/02/16 08:28:35 INFO mapred.JobClient:  map 0% reduce 0%
>> 10/02/16 08:28:51 INFO mapred.JobClient:  map 20% reduce 0%
>> 10/02/16 08:29:00 INFO mapred.JobClient:  map 40% reduce 0%
>> 10/02/16 08:29:06 INFO mapred.JobClient:  map 40% reduce 1%
>> 10/02/16 08:29:09 INFO mapred.JobClient:  map 60% reduce 2%
>> 10/02/16 08:29:15 INFO mapred.JobClient:  map 80% reduce 2%
>> 10/02/16 08:29:18 INFO mapred.JobClient:  map 80% reduce 3%
>> 10/02/16 08:29:21 INFO mapred.JobClient:  map 100% reduce 4%
>> 10/02/16 08:29:24 INFO mapred.JobClient:  map 100% reduce 5%
>> 10/02/16 08:29:27 INFO mapred.JobClient:  map 100% reduce 12%
>> 10/02/16 08:29:30 INFO mapred.JobClient:  map 100% reduce 20%
>> 10/02/16 08:29:36 INFO mapred.JobClient:  map 100% reduce 23%
>> 10/02/16 08:29:39 INFO mapred.JobClient:  map 100% reduce 33%
>> 10/02/16 08:29:42 INFO mapred.JobClient:  map 100% reduce 40%
>> 10/02/16 08:29:48 INFO mapred.JobClient:  map 100% reduce 43%
>> 10/02/16 08:29:51 INFO mapred.JobClient:  map 100% reduce 60%
>> 10/02/16 08:30:01 INFO mapred.JobClient:  map 100% reduce 70%
>> 10/02/16 08:30:04 INFO mapred.JobClient:  map 100% reduce 73%
>> 10/02/16 08:30:07 INFO mapred.JobClient:  map 100% reduce 80%
>> 10/02/16 08:30:12 INFO mapred.JobClient:  map 100% reduce 100%
>> 10/02/16 08:30:14 INFO mapred.JobClient: Job complete: job_201002160823_0001
>> 10/02/16 08:30:14 INFO mapred.JobClient: Counters: 16
>> 10/02/16 08:30:14 INFO mapred.JobClient:   Job Counters
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Launched reduce tasks=10
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Launched map tasks=10
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Data-local map tasks=10
>> 10/02/16 08:30:14 INFO mapred.JobClient:   FileSystemCounters
>> 10/02/16 08:30:14 INFO mapred.JobClient:     FILE_BYTES_READ=42372
>> 10/02/16 08:30:14 INFO mapred.JobClient:     HDFS_BYTES_READ=1080
>> 10/02/16 08:30:14 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=87764
>> 10/02/16 08:30:14 INFO mapred.JobClient:   Map-Reduce Framework
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce input groups=100
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Combine output records=0
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Map input records=10
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce shuffle bytes=42463
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce output records=100
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Spilled Records=200
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Map output bytes=41912
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Combine input records=0
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Map output records=100
>> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce input records=100
>> 10/02/16 08:30:14 INFO zookeeper.ZooKeeper: Closing session:
>> 0x126d6f5b8a90003
>> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session:
>> 0x126d6f5b8a90003
>> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Exception while closing send
>> thread for session 0x126d6f5b8a90003 : Read error rc = -1
>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> 10/02/16 08:30:14 WARN zookeeper.ClientCnxn: Ignoring exception during
>> shutdown input
>> java.net.SocketException: Socket is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
>> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for
>> session: 0x126d6f5b8a90003
>> 10/02/16 08:30:14 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a90003
>> closed
>> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: EventThread shut down
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:host.name
>> =192.168.0.11
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.version=1.6.0_17
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.vendor=Apple Inc.
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:java.compiler=<NA>
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
>> OS X
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:os.arch=x86_64
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:os.version=10.6.2
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:user.name
>> =hadoop
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:user.home=/Users/hadoop
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
>> environment:user.dir=/opt/local/hadoop/jobs/hama
>> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Initiating client connection,
>> connectString=localhost:2181 sessionTimeout=60000
>> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@2efb56b1
>> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
>> is false
>> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Attempting connection to server
>> localhost/fe80:0:0:0:0:0:0:1%1:2181
>> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Priming connection to
>> java.nio.channels.SocketChannel[connected local=/fe80:0:0:0:0:0:0:1%1:58457
>> remote=localhost/fe80:0:0:0:0:0:0:1%1:2181]
>> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Server connection successful
>> 10/02/16 08:30:17 INFO matrix.AbstractMatrix: Initializing the matrix
>> storage.
>> 10/02/16 08:30:19 INFO matrix.AbstractMatrix: Create Matrix
>> SparseMatrix_randflbeh
>> 10/02/16 08:30:19 INFO matrix.AbstractMatrix: Create the 100 * 100 random
>> matrix : SparseMatrix_randflbeh
>> Wrote input for Map #0
>> Wrote input for Map #1
>> Wrote input for Map #2
>> Wrote input for Map #3
>> Wrote input for Map #4
>> Wrote input for Map #5
>> Wrote input for Map #6
>> Wrote input for Map #7
>> Wrote input for Map #8
>> Wrote input for Map #9
>> 10/02/16 08:30:20 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 10/02/16 08:30:20 WARN mapred.JobClient: No job jar file set.  User classes
>> may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 10/02/16 08:30:20 INFO input.FileInputFormat: Total input paths to process :
>> 10
>> 10/02/16 08:30:21 INFO mapred.JobClient: Running job: job_201002160823_0002
>> 10/02/16 08:30:22 INFO mapred.JobClient:  map 0% reduce 0%
>> 10/02/16 08:30:33 INFO mapred.JobClient:  map 20% reduce 0%
>> 10/02/16 08:30:42 INFO mapred.JobClient:  map 40% reduce 0%
>> 10/02/16 08:30:45 INFO mapred.JobClient:  map 40% reduce 1%
>> 10/02/16 08:30:51 INFO mapred.JobClient:  map 60% reduce 2%
>> 10/02/16 08:30:57 INFO mapred.JobClient:  map 80% reduce 2%
>> 10/02/16 08:31:00 INFO mapred.JobClient:  map 80% reduce 4%
>> 10/02/16 08:31:03 INFO mapred.JobClient:  map 100% reduce 4%
>> 10/02/16 08:31:06 INFO mapred.JobClient:  map 100% reduce 12%
>> 10/02/16 08:31:12 INFO mapred.JobClient:  map 100% reduce 30%
>> 10/02/16 08:31:22 INFO mapred.JobClient:  map 100% reduce 43%
>> 10/02/16 08:31:25 INFO mapred.JobClient:  map 100% reduce 50%
>> 10/02/16 08:31:31 INFO mapred.JobClient:  map 100% reduce 53%
>> 10/02/16 08:31:34 INFO mapred.JobClient:  map 100% reduce 63%
>> 10/02/16 08:31:37 INFO mapred.JobClient:  map 100% reduce 70%
>> 10/02/16 08:31:43 INFO mapred.JobClient:  map 100% reduce 73%
>> 10/02/16 08:31:46 INFO mapred.JobClient:  map 100% reduce 83%
>> 10/02/16 08:31:49 INFO mapred.JobClient:  map 100% reduce 90%
>> 10/02/16 08:31:55 INFO mapred.JobClient:  map 100% reduce 93%
>> 10/02/16 08:31:58 INFO mapred.JobClient:  map 100% reduce 100%
>> 10/02/16 08:32:00 INFO mapred.JobClient: Job complete: job_201002160823_0002
>> 10/02/16 08:32:00 INFO mapred.JobClient: Counters: 16
>> 10/02/16 08:32:00 INFO mapred.JobClient:   Job Counters
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Launched reduce tasks=10
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Launched map tasks=10
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Data-local map tasks=10
>> 10/02/16 08:32:00 INFO mapred.JobClient:   FileSystemCounters
>> 10/02/16 08:32:00 INFO mapred.JobClient:     FILE_BYTES_READ=42512
>> 10/02/16 08:32:00 INFO mapred.JobClient:     HDFS_BYTES_READ=1080
>> 10/02/16 08:32:00 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=88044
>> 10/02/16 08:32:00 INFO mapred.JobClient:   Map-Reduce Framework
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce input groups=100
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Combine output records=0
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Map input records=10
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce shuffle bytes=42603
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce output records=100
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Spilled Records=200
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Map output bytes=42052
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Combine input records=0
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Map output records=100
>> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce input records=100
>> 10/02/16 08:32:00 INFO zookeeper.ZooKeeper: Closing session:
>> 0x126d6f5b8a9000e
>> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session:
>> 0x126d6f5b8a9000e
>> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Exception while closing send
>> thread for session 0x126d6f5b8a9000e : Read error rc = -1
>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> 10/02/16 08:32:00 WARN zookeeper.ClientCnxn: Ignoring exception during
>> shutdown input
>> java.net.SocketException: Socket is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
>> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for
>> session: 0x126d6f5b8a9000e
>> 10/02/16 08:32:00 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a9000e
>> closed
>> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: EventThread shut down
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:host.name
>> =192.168.0.11
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.version=1.6.0_17
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.vendor=Apple Inc.
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:java.compiler=<NA>
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
>> OS X
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:os.arch=x86_64
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:os.version=10.6.2
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:user.name
>> =hadoop
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:user.home=/Users/hadoop
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
>> environment:user.dir=/opt/local/hadoop/jobs/hama
>> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Initiating client connection,
>> connectString=localhost:2181 sessionTimeout=60000
>> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@76f8968f
>> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
>> is false
>> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Attempting connection to server
>> localhost/0:0:0:0:0:0:0:1:2181
>> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Priming connection to
>> java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:58679
>> remote=localhost/0:0:0:0:0:0:0:1:2181]
>> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Server connection successful
>> C = A + B
>> 10/02/16 08:32:05 INFO matrix.AbstractMatrix: Initializing the matrix
>> storage.
>> 10/02/16 08:32:07 INFO matrix.AbstractMatrix: Create Matrix
>> DenseMatrix_randwhqtl
>> 10/02/16 08:32:07 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 10/02/16 08:32:07 WARN mapred.JobClient: No job jar file set.  User classes
>> may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 10/02/16 08:32:08 INFO mapred.JobClient: Running job: job_201002160823_0003
>> 10/02/16 08:32:09 INFO mapred.JobClient:  map 0% reduce 0%
>> 10/02/16 08:32:23 INFO mapred.JobClient: Task Id :
>> attempt_201002160823_0003_m_000000_0, Status : FAILED
>> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
>> at
>> org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92)
>>  at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150)
>> at
>> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35)
>>  at
>> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 10/02/16 08:32:33 INFO mapred.JobClient: Task Id :
>> attempt_201002160823_0003_m_000000_1, Status : FAILED
>> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
>> at
>> org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92)
>>  at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150)
>> at
>> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35)
>>  at
>> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 10/02/16 08:32:42 INFO mapred.JobClient: Task Id :
>> attempt_201002160823_0003_m_000000_2, Status : FAILED
>> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
>> at
>> org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92)
>>  at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150)
>> at
>> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35)
>>  at
>> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 10/02/16 08:32:54 INFO mapred.JobClient: Job complete: job_201002160823_0003
>> 10/02/16 08:32:54 INFO mapred.JobClient: Counters: 3
>> 10/02/16 08:32:54 INFO mapred.JobClient:   Job Counters
>> 10/02/16 08:32:54 INFO mapred.JobClient:     Launched map tasks=4
>> 10/02/16 08:32:54 INFO mapred.JobClient:     Data-local map tasks=4
>> 10/02/16 08:32:54 INFO mapred.JobClient:     Failed map tasks=1
>> 10/02/16 08:32:56 INFO client.HBaseAdmin: Disabled DenseMatrix_randwhqtl
>> 10/02/16 08:32:56 INFO client.HBaseAdmin: Deleted DenseMatrix_randwhqtl
>> 10/02/16 08:32:56 INFO zookeeper.ZooKeeper: Closing session:
>> 0x126d6f5b8a90019
>> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session:
>> 0x126d6f5b8a90019
>> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Exception while closing send
>> thread for session 0x126d6f5b8a90019 : Read error rc = -1
>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
>> 10/02/16 08:32:56 WARN zookeeper.ClientCnxn: Ignoring exception during
>> shutdown input
>> java.net.SocketException: Socket is not connected
>> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
>> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for
>> session: 0x126d6f5b8a90019
>> 10/02/16 08:32:56 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a90019
>> closed
>> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: EventThread shut down
>>  run (rand, rand, add)
>>
>
>
>
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> edwardyoon@apache.org
> http://blog.udanax.org
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message