hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: How can I run hama in eclipse environment
Date Fri, 25 Apr 2014 09:55:56 GMT
Very strange. Sorry, I'm usually not use Eclipse. Have you tried to
increase the max heap size?

Run configurations -> Arguments -> VM arguments -> -Xmx1024m

Or, alternatively, you can run daemons using terminals:

$ bin/hama zookeeper
$ bin/hama bspmaster
$ bin/hama groom

or

$ bin/start-bspd.sh

Then, you'll be able to launch a BSP job from eclipse.

On Fri, Apr 25, 2014 at 6:35 PM, InJun Song <ijsong@gmail.com> wrote:
> Thanks for your reply.
>
> I am running hadoop pseudo distributed mode. (apache hadoop 1.2.1)
>
> When I run HAMA 0.6.4 release in pseudo distributed mode (using
> start-bspd.sh) and execute my dummy bsp job in hama commands (hama jar
> ...), it works well.
>
> The problem happens when I ran daemons and my job in eclipse.
>
> Below is my full log of BSPMasterRunner in eclipse console.
>
>
>
> 14/04/25 18:29:01 INFO hama.BSPMasterRunner: STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting BSPMaster
>
> STARTUP_MSG:   host = km-knuth.local/172.29.9.73
>
> STARTUP_MSG:   args = []
>
> STARTUP_MSG:   version = 1.2.0
>
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r
> 1479473; compiled by 'hortonfo' on Mon May  6 18:29:07 UTC 2013
>
> STARTUP_MSG:   java = 1.6.0_65
>
> ************************************************************/
>
> 14/04/25 18:29:01 INFO bsp.BSPMaster: RPC BSPMaster: host localhost port
> 40000
>
> 2014-04-25 18:29:01.614 java[53927:1003] Unable to load realm info from
> SCDynamicStore
>
> 14/04/25 18:29:01 INFO ipc.Server: Starting Socket Reader #1 for port 40000
>
> 14/04/25 18:29:01 INFO mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
>
> 14/04/25 18:29:01 INFO http.HttpServer: Port returned by
> webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
> the listener on 40013
>
> 14/04/25 18:29:01 INFO http.HttpServer: listener.getLocalPort() returned
> 40013 webServer.getConnectors()[0].getLocalPort() returned 40013
>
> 14/04/25 18:29:01 INFO http.HttpServer: Jetty bound to port 40013
>
> 14/04/25 18:29:01 INFO mortbay.log: jetty-6.1.14
>
> 14/04/25 18:29:01 INFO mortbay.log: Started SelectChannelConnector@localhost
> :40013
>
> 14/04/25 18:29:02 INFO bsp.BSPMaster: Cleaning up the system directory
>
> 14/04/25 18:29:02 INFO bsp.BSPMaster:
> hdfs://localhost:9000/tmp/hadoop-juns/bsp/system
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client environment:host.name
> =172.29.9.73
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_65
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Apple Inc.
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/Users/juns/workspaces/private/hama/core/target/test-classes:/Users/juns/workspaces/private/hama/core/target/classes:/Users/juns/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/juns/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/juns/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/juns/workspaces/private/hama/commons/target/test-classes:/Users/juns/workspaces/private/hama/commons/target/classes:/Users/juns/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.jar:/Users/juns/.m2/repository/org/apache/hadoop/hadoop-core/1.2.0/hadoop-core-1.2.0.jar:/Users/juns/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/juns/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/juns/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/Users/juns/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/Users/juns/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/juns/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/juns/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.2/jackson-core-asl-1.9.2.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.2/jackson-mapper-asl-1.9.2.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.7.1/jackson-jaxrs-1.7.1.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar:/Users/juns/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/Users/juns/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/juns/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/Users/juns/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:/Users/juns/.m2/repository/junit/junit/4.8.1/junit-4.8.1.jar:/Users/juns/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/juns/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/juns/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/juns/.m2/repository/commons-configuration/commons-configuration/1.7/commons-configuration-1.7.jar:/Users/juns/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/juns/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/juns/.m2/repository/commons-digester/commons-digester/1.8.1/commons-digester-1.8.1.jar:/Users/juns/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar:/Users/juns/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/juns/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty/6.1.14/jetty-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty-util/6.1.14/jetty-util-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/Users/juns/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/juns/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/Users/juns/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/Users/juns/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar:/Users/juns/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/juns/.m2/repository/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar:/Users/juns/.m2/repository/hsqldb/hsqldb/
> 1.8.0.10/hsqldb-1.8.0.10.jar:/Users/juns/.m2/repository/org/apache/hadoop/hadoop-test/1.2.0/hadoop-test-1.2.0.jar:/Users/juns/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/Users/juns/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/Users/juns/.m2/repository/org/slf4j/slf4j-api/1.5.8/slf4j-api-1.5.8.jar:/Users/juns/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/Users/juns/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/Users/juns/.m2/repository/commons-codec/commons-codec/1.2/commons-codec-1.2.jar:/Users/juns/.m2/repository/org/xerial/snappy/snappy-java/1.0.5/snappy-java-1.0.5.jar:/Users/juns/.m2/repository/org/apache/commons/commons-math3/3.0/commons-math3-3.0.jar:/Users/juns/.m2/repository/org/apache/ant/ant/1.7.1/ant-1.7.1.jar:/Users/juns/.m2/repository/org/apache/ant/ant-launcher/1.7.1/ant-launcher-1.7.1.jar:/Users/juns/.m2/repository/org/apache/tomcat/servlet-api/6.0.32/servlet-api-6.0.32.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty-annotations/6.1.14/jetty-annotations-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty-plus/6.1.14/jetty-plus-6.1.14.jar:/Users/juns/.m2/repository/geronimo-spec/geronimo-spec-jta/1.0.1B-rc4/geronimo-spec-jta-1.0.1B-rc4.jar:/Users/juns/.m2/repository/org/apache/geronimo/specs/geronimo-annotation_1.0_spec/1.0/geronimo-annotation_1.0_spec-1.0.jar:/Users/juns/.m2/repository/org/slf4j/slf4j-log4j12/1.5.8/slf4j-log4j12-1.5.8.jar:/Users/juns/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/Users/juns/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/juns/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/juns/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar:/Users/juns/workspaces/private/hama/conf
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/var/folders/rv/42slk6553x7885qkq08kdt3r0000gn/T/
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac
> OS X
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=x86_64
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:os.version=10.9.2
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client environment:user.name
> =juns
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/Users/juns
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/Users/juns/workspaces/private/hama/core
>
> 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=localhost:21810 sessionTimeout=1200000
> watcher=org.apache.hama.bsp.sync.ZKSyncBSPMasterClient@76d78df0
>
> 14/04/25 18:29:02 INFO sync.ZKSyncBSPMasterClient: Initialized ZK false
>
> 14/04/25 18:29:02 INFO sync.ZKSyncClient: Initializing ZK Sync Client
>
> 14/04/25 18:29:02 INFO zookeeper.ClientCnxn: Opening socket connection to
> server localhost/0:0:0:0:0:0:0:1:21810. Will not attempt to authenticate
> using SASL (Unable to locate a login configuration)
>
> 14/04/25 18:29:02 INFO zookeeper.ClientCnxn: Socket connection established
> to localhost/0:0:0:0:0:0:0:1:21810, initiating session
>
> 14/04/25 18:29:02 INFO zookeeper.ClientCnxn: Session establishment complete
> on server localhost/0:0:0:0:0:0:0:1:21810, sessionid = 0x145983756eb0000,
> negotiated timeout = 1200000
>
> 14/04/25 18:29:02 INFO ipc.Server: IPC Server Responder: starting
>
> 14/04/25 18:29:02 INFO ipc.Server: IPC Server listener on 40000: starting
>
> 14/04/25 18:29:02 INFO ipc.Server: IPC Server handler 0 on 40000: starting
>
> 14/04/25 18:29:02 INFO bsp.BSPMaster: Starting RUNNING
>
> 14/04/25 18:29:05 INFO bsp.BSPMaster: groomd_172.29.9.73_50000 is added.
>
> Exception in thread "Socket Reader #1 for port 40000"
> java.lang.OutOfMemoryError: Java heap space
>
> at org.apache.hama.ipc.RPC$Invocation.readFields(RPC.java:111)
>
> at org.apache.hama.ipc.Server$Connection.processData(Server.java:1121)
>
> at org.apache.hama.ipc.Server$Connection.processOneRpc(Server.java:1099)
>
> at org.apache.hama.ipc.Server$Connection.readAndProcess(Server.java:1063)
>
> at org.apache.hama.ipc.Server$Listener.doRead(Server.java:576)
>
>  at org.apache.hama.ipc.Server$Listener$Reader.run(Server.java:363)
>
>
> On Fri, Apr 25, 2014 at 4:57 PM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>
>> Hi, can you attach the full log of the BSPMaster?
>>
>> Just to be sure, to run Hama in pseudo distributed mode, you must run
>> HDFS first.
>>
>> On Fri, Apr 25, 2014 at 4:19 PM, InJun Song <ijsong@gmail.com> wrote:
>> > Hi,
>> >
>> > I have a question about how to run hama in eclipse environment.
>> >
>> > At first, I set up my develop environment according to this page. (
>> > http://wiki.apache.org/hama/HowToContribute)
>> >
>> > Then, because I wanted to run pseudo distributed mode, I ran each daemon
>> > under eclipse. (zookeeper -> bspmaster -> groomserver). To run those,
I
>> > executed three runner for them; ZookeeprRunner, BSPMasterRunner and
>> > GroomserverRunner.
>> > After all daemons launched, I ran dummy BSP program.
>> >
>> > At that time, BSPMaster throws OOM exception. Code throws an exception is
>> > in "RPC.Invocation.readFields" method. While it executed this line, it
>> > throws the exception.
>> >
>> >     parameters = new Object[in.readInt()];
>> >
>> > The number of object to create is 308766068. If sizeof(java.lang.Object)
>> is
>> > 8 byte, total size is over 2G. I think that it must be wrong.
>> >
>> > What did I mistake? What mistake make me go wrong way?
>> > I am debugging with eclipse, but I want to some advices from you.
>> >
>> > Thank you.
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
>>



-- 
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.

Mime
View raw message