incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Hama "add, multi, norms, similarity" examples problems
Date Tue, 16 Feb 2010 14:07:40 GMT
Hi, Andrew

>> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)

Eh, It seems an un-expected exception. Let me check it out.

>> Another question is about lack of "breadth first search" sample,
>> is this a sub-project of next release?

Using M/R iterations, you can simply implement it --
http://blog.udanax.org/2009/02/breadth-first-search-mapreduce.html

But, an M/R iterative job is *awfully* slow, so I think it's not worth
to implement.

Instead, we trying to implement a BSP-based graph computing framework
to improve the performance of graph algorithms on Hadoop.

On Tue, Feb 16, 2010 at 10:41 PM, Andrew Milkowski
<andrewmilkowski@gmail.com> wrote:
> Hi there
>
> Have following working setup:
>
> hadoop 0.20.1
> hbase 0.20.3
> hama latest SVN trunk version
>
> Notes:
>
> had to replace and rebuild hama with new hbase lib (upgraded to version
> 0.20.3)
>
> 1. successfully rand hama example to generate random matrix
>
> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA
>
> Note: verified hbase table was generated (via shell list and scan)
>
> 2. moved on to other examples, generated 2 random matrices:
>
> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA
> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixB
>
> then an attempt at distributed addition:
>
> $HAMA_HOME/bin/hama examples add1  matrixA matrixB
>
>
> 3. in similar venue tried multi, norms and and similarity (all generated
> exceptions)
>
> Exceptions appear to be hama code related (and not environmentally specific)
>
> Culprit could be in improper usage of input matrix data to add,multi, norms
> and similarity.
>
> Another question is about lack of "breadth first search" sample, is this a
> sub-project of next release?
>
> thanks
>
> below attached console stack trace of bash-3.2$ ./run_examples.sh (Note:
> ignore  INFO zookeeper.ClientCnxn: Exception while closing send thread for
> session... exception that is a known "INFO" level exception present in
> zookeeper client/server socket code fragment)
>
>
> --------------------------------------------
>
> run_examples.sh:
>
> #!/bin/bash
>
> export HAMA_JAR="${HAMA_HOME}/hama-0.2.0-dev-examples.jar"
>
> # cygwin path translation
> #if $cygwin; then
> #  HAMA_CLASSPATH=`cygpath -p -w "$HAMA_CLASSPATH"`
> #fi
>
> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA
>
> $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixB
>
> $HAMA_HOME/bin/hama examples add matrixA matrixB
>
> exit $?
>
>
>
>
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:host.name
> =192.168.0.11
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_17
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Apple Inc.
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
> OS X
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=x86_64
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:os.version=10.6.2
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:user.name
> =hadoop
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/Users/hadoop
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/local/hadoop/jobs/hama
> 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=localhost:2181 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@2efb56b1
> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
> is false
> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Attempting connection to server
> localhost/127.0.0.1:2181
> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.channels.SocketChannel[connected
> local=/127.0.0.1:58238remote=localhost/
> 127.0.0.1:2181]
> 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Server connection successful
> 10/02/16 08:28:30 INFO matrix.AbstractMatrix: Initializing the matrix
> storage.
> 10/02/16 08:28:32 INFO matrix.AbstractMatrix: Create Matrix
> SparseMatrix_randliqns
> 10/02/16 08:28:32 INFO matrix.AbstractMatrix: Create the 100 * 100 random
> matrix : SparseMatrix_randliqns
> Wrote input for Map #0
> Wrote input for Map #1
> Wrote input for Map #2
> Wrote input for Map #3
> Wrote input for Map #4
> Wrote input for Map #5
> Wrote input for Map #6
> Wrote input for Map #7
> Wrote input for Map #8
> Wrote input for Map #9
> 10/02/16 08:28:33 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/02/16 08:28:33 WARN mapred.JobClient: No job jar file set.  User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 10/02/16 08:28:33 INFO input.FileInputFormat: Total input paths to process :
> 10
> 10/02/16 08:28:34 INFO mapred.JobClient: Running job: job_201002160823_0001
> 10/02/16 08:28:35 INFO mapred.JobClient:  map 0% reduce 0%
> 10/02/16 08:28:51 INFO mapred.JobClient:  map 20% reduce 0%
> 10/02/16 08:29:00 INFO mapred.JobClient:  map 40% reduce 0%
> 10/02/16 08:29:06 INFO mapred.JobClient:  map 40% reduce 1%
> 10/02/16 08:29:09 INFO mapred.JobClient:  map 60% reduce 2%
> 10/02/16 08:29:15 INFO mapred.JobClient:  map 80% reduce 2%
> 10/02/16 08:29:18 INFO mapred.JobClient:  map 80% reduce 3%
> 10/02/16 08:29:21 INFO mapred.JobClient:  map 100% reduce 4%
> 10/02/16 08:29:24 INFO mapred.JobClient:  map 100% reduce 5%
> 10/02/16 08:29:27 INFO mapred.JobClient:  map 100% reduce 12%
> 10/02/16 08:29:30 INFO mapred.JobClient:  map 100% reduce 20%
> 10/02/16 08:29:36 INFO mapred.JobClient:  map 100% reduce 23%
> 10/02/16 08:29:39 INFO mapred.JobClient:  map 100% reduce 33%
> 10/02/16 08:29:42 INFO mapred.JobClient:  map 100% reduce 40%
> 10/02/16 08:29:48 INFO mapred.JobClient:  map 100% reduce 43%
> 10/02/16 08:29:51 INFO mapred.JobClient:  map 100% reduce 60%
> 10/02/16 08:30:01 INFO mapred.JobClient:  map 100% reduce 70%
> 10/02/16 08:30:04 INFO mapred.JobClient:  map 100% reduce 73%
> 10/02/16 08:30:07 INFO mapred.JobClient:  map 100% reduce 80%
> 10/02/16 08:30:12 INFO mapred.JobClient:  map 100% reduce 100%
> 10/02/16 08:30:14 INFO mapred.JobClient: Job complete: job_201002160823_0001
> 10/02/16 08:30:14 INFO mapred.JobClient: Counters: 16
> 10/02/16 08:30:14 INFO mapred.JobClient:   Job Counters
> 10/02/16 08:30:14 INFO mapred.JobClient:     Launched reduce tasks=10
> 10/02/16 08:30:14 INFO mapred.JobClient:     Launched map tasks=10
> 10/02/16 08:30:14 INFO mapred.JobClient:     Data-local map tasks=10
> 10/02/16 08:30:14 INFO mapred.JobClient:   FileSystemCounters
> 10/02/16 08:30:14 INFO mapred.JobClient:     FILE_BYTES_READ=42372
> 10/02/16 08:30:14 INFO mapred.JobClient:     HDFS_BYTES_READ=1080
> 10/02/16 08:30:14 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=87764
> 10/02/16 08:30:14 INFO mapred.JobClient:   Map-Reduce Framework
> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce input groups=100
> 10/02/16 08:30:14 INFO mapred.JobClient:     Combine output records=0
> 10/02/16 08:30:14 INFO mapred.JobClient:     Map input records=10
> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce shuffle bytes=42463
> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce output records=100
> 10/02/16 08:30:14 INFO mapred.JobClient:     Spilled Records=200
> 10/02/16 08:30:14 INFO mapred.JobClient:     Map output bytes=41912
> 10/02/16 08:30:14 INFO mapred.JobClient:     Combine input records=0
> 10/02/16 08:30:14 INFO mapred.JobClient:     Map output records=100
> 10/02/16 08:30:14 INFO mapred.JobClient:     Reduce input records=100
> 10/02/16 08:30:14 INFO zookeeper.ZooKeeper: Closing session:
> 0x126d6f5b8a90003
> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session:
> 0x126d6f5b8a90003
> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Exception while closing send
> thread for session 0x126d6f5b8a90003 : Read error rc = -1
> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 10/02/16 08:30:14 WARN zookeeper.ClientCnxn: Ignoring exception during
> shutdown input
> java.net.SocketException: Socket is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for
> session: 0x126d6f5b8a90003
> 10/02/16 08:30:14 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a90003
> closed
> 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: EventThread shut down
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:host.name
> =192.168.0.11
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_17
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Apple Inc.
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
> OS X
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=x86_64
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:os.version=10.6.2
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:user.name
> =hadoop
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/Users/hadoop
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/local/hadoop/jobs/hama
> 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=localhost:2181 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@2efb56b1
> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
> is false
> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Attempting connection to server
> localhost/fe80:0:0:0:0:0:0:1%1:2181
> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.channels.SocketChannel[connected local=/fe80:0:0:0:0:0:0:1%1:58457
> remote=localhost/fe80:0:0:0:0:0:0:1%1:2181]
> 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Server connection successful
> 10/02/16 08:30:17 INFO matrix.AbstractMatrix: Initializing the matrix
> storage.
> 10/02/16 08:30:19 INFO matrix.AbstractMatrix: Create Matrix
> SparseMatrix_randflbeh
> 10/02/16 08:30:19 INFO matrix.AbstractMatrix: Create the 100 * 100 random
> matrix : SparseMatrix_randflbeh
> Wrote input for Map #0
> Wrote input for Map #1
> Wrote input for Map #2
> Wrote input for Map #3
> Wrote input for Map #4
> Wrote input for Map #5
> Wrote input for Map #6
> Wrote input for Map #7
> Wrote input for Map #8
> Wrote input for Map #9
> 10/02/16 08:30:20 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/02/16 08:30:20 WARN mapred.JobClient: No job jar file set.  User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 10/02/16 08:30:20 INFO input.FileInputFormat: Total input paths to process :
> 10
> 10/02/16 08:30:21 INFO mapred.JobClient: Running job: job_201002160823_0002
> 10/02/16 08:30:22 INFO mapred.JobClient:  map 0% reduce 0%
> 10/02/16 08:30:33 INFO mapred.JobClient:  map 20% reduce 0%
> 10/02/16 08:30:42 INFO mapred.JobClient:  map 40% reduce 0%
> 10/02/16 08:30:45 INFO mapred.JobClient:  map 40% reduce 1%
> 10/02/16 08:30:51 INFO mapred.JobClient:  map 60% reduce 2%
> 10/02/16 08:30:57 INFO mapred.JobClient:  map 80% reduce 2%
> 10/02/16 08:31:00 INFO mapred.JobClient:  map 80% reduce 4%
> 10/02/16 08:31:03 INFO mapred.JobClient:  map 100% reduce 4%
> 10/02/16 08:31:06 INFO mapred.JobClient:  map 100% reduce 12%
> 10/02/16 08:31:12 INFO mapred.JobClient:  map 100% reduce 30%
> 10/02/16 08:31:22 INFO mapred.JobClient:  map 100% reduce 43%
> 10/02/16 08:31:25 INFO mapred.JobClient:  map 100% reduce 50%
> 10/02/16 08:31:31 INFO mapred.JobClient:  map 100% reduce 53%
> 10/02/16 08:31:34 INFO mapred.JobClient:  map 100% reduce 63%
> 10/02/16 08:31:37 INFO mapred.JobClient:  map 100% reduce 70%
> 10/02/16 08:31:43 INFO mapred.JobClient:  map 100% reduce 73%
> 10/02/16 08:31:46 INFO mapred.JobClient:  map 100% reduce 83%
> 10/02/16 08:31:49 INFO mapred.JobClient:  map 100% reduce 90%
> 10/02/16 08:31:55 INFO mapred.JobClient:  map 100% reduce 93%
> 10/02/16 08:31:58 INFO mapred.JobClient:  map 100% reduce 100%
> 10/02/16 08:32:00 INFO mapred.JobClient: Job complete: job_201002160823_0002
> 10/02/16 08:32:00 INFO mapred.JobClient: Counters: 16
> 10/02/16 08:32:00 INFO mapred.JobClient:   Job Counters
> 10/02/16 08:32:00 INFO mapred.JobClient:     Launched reduce tasks=10
> 10/02/16 08:32:00 INFO mapred.JobClient:     Launched map tasks=10
> 10/02/16 08:32:00 INFO mapred.JobClient:     Data-local map tasks=10
> 10/02/16 08:32:00 INFO mapred.JobClient:   FileSystemCounters
> 10/02/16 08:32:00 INFO mapred.JobClient:     FILE_BYTES_READ=42512
> 10/02/16 08:32:00 INFO mapred.JobClient:     HDFS_BYTES_READ=1080
> 10/02/16 08:32:00 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=88044
> 10/02/16 08:32:00 INFO mapred.JobClient:   Map-Reduce Framework
> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce input groups=100
> 10/02/16 08:32:00 INFO mapred.JobClient:     Combine output records=0
> 10/02/16 08:32:00 INFO mapred.JobClient:     Map input records=10
> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce shuffle bytes=42603
> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce output records=100
> 10/02/16 08:32:00 INFO mapred.JobClient:     Spilled Records=200
> 10/02/16 08:32:00 INFO mapred.JobClient:     Map output bytes=42052
> 10/02/16 08:32:00 INFO mapred.JobClient:     Combine input records=0
> 10/02/16 08:32:00 INFO mapred.JobClient:     Map output records=100
> 10/02/16 08:32:00 INFO mapred.JobClient:     Reduce input records=100
> 10/02/16 08:32:00 INFO zookeeper.ZooKeeper: Closing session:
> 0x126d6f5b8a9000e
> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session:
> 0x126d6f5b8a9000e
> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Exception while closing send
> thread for session 0x126d6f5b8a9000e : Read error rc = -1
> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 10/02/16 08:32:00 WARN zookeeper.ClientCnxn: Ignoring exception during
> shutdown input
> java.net.SocketException: Socket is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for
> session: 0x126d6f5b8a9000e
> 10/02/16 08:32:00 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a9000e
> closed
> 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: EventThread shut down
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:host.name
> =192.168.0.11
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_17
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Apple Inc.
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
> OS X
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=x86_64
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:os.version=10.6.2
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:user.name
> =hadoop
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/Users/hadoop
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/local/hadoop/jobs/hama
> 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=localhost:2181 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@76f8968f
> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
> is false
> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Attempting connection to server
> localhost/0:0:0:0:0:0:0:1:2181
> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:58679
> remote=localhost/0:0:0:0:0:0:0:1:2181]
> 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Server connection successful
> C = A + B
> 10/02/16 08:32:05 INFO matrix.AbstractMatrix: Initializing the matrix
> storage.
> 10/02/16 08:32:07 INFO matrix.AbstractMatrix: Create Matrix
> DenseMatrix_randwhqtl
> 10/02/16 08:32:07 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/02/16 08:32:07 WARN mapred.JobClient: No job jar file set.  User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 10/02/16 08:32:08 INFO mapred.JobClient: Running job: job_201002160823_0003
> 10/02/16 08:32:09 INFO mapred.JobClient:  map 0% reduce 0%
> 10/02/16 08:32:23 INFO mapred.JobClient: Task Id :
> attempt_201002160823_0003_m_000000_0, Status : FAILED
> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
> at
> org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92)
>  at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150)
> at
> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35)
>  at
> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 10/02/16 08:32:33 INFO mapred.JobClient: Task Id :
> attempt_201002160823_0003_m_000000_1, Status : FAILED
> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
> at
> org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92)
>  at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150)
> at
> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35)
>  at
> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 10/02/16 08:32:42 INFO mapred.JobClient: Task Id :
> attempt_201002160823_0003_m_000000_2, Status : FAILED
> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
> at
> org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92)
>  at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150)
> at
> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35)
>  at
> org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 10/02/16 08:32:54 INFO mapred.JobClient: Job complete: job_201002160823_0003
> 10/02/16 08:32:54 INFO mapred.JobClient: Counters: 3
> 10/02/16 08:32:54 INFO mapred.JobClient:   Job Counters
> 10/02/16 08:32:54 INFO mapred.JobClient:     Launched map tasks=4
> 10/02/16 08:32:54 INFO mapred.JobClient:     Data-local map tasks=4
> 10/02/16 08:32:54 INFO mapred.JobClient:     Failed map tasks=1
> 10/02/16 08:32:56 INFO client.HBaseAdmin: Disabled DenseMatrix_randwhqtl
> 10/02/16 08:32:56 INFO client.HBaseAdmin: Deleted DenseMatrix_randwhqtl
> 10/02/16 08:32:56 INFO zookeeper.ZooKeeper: Closing session:
> 0x126d6f5b8a90019
> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session:
> 0x126d6f5b8a90019
> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Exception while closing send
> thread for session 0x126d6f5b8a90019 : Read error rc = -1
> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 10/02/16 08:32:56 WARN zookeeper.ClientCnxn: Ignoring exception during
> shutdown input
> java.net.SocketException: Socket is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for
> session: 0x126d6f5b8a90019
> 10/02/16 08:32:56 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a90019
> closed
> 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: EventThread shut down
>  run (rand, rand, add)
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message