hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From InJun Song <ijs...@gmail.com>
Subject Re: How can I run hama in eclipse environment
Date Fri, 25 Apr 2014 13:02:58 GMT
In case of running hama separately like "bin/start-bspd.sh" or "bin/hama
zookeeper && bin/hama bspmaster && bin/hama groom", executing bsp job in
eclipse works well.

But, running all hama daemons in eclipse doesn't work.
Although heap size is configured higher (up to 4G), strange exceptions are
thrown.
I think that executing hama processes in eclipse IDE results in abnormal
problems.

I'd better find other way to run pseudo distributed hama in debug mode. I
searched past Q&A in hama archive;
  - https://www.mail-archive.com/hama-dev@incubator.apache.org/msg04587.html
  - https://www.mail-archive.com/hama-dev@incubator.apache.org/msg04588.html

Very thanks for your help, Edward!



On Fri, Apr 25, 2014 at 6:55 PM, Edward J. Yoon <edwardyoon@apache.org>wrote:

> Very strange. Sorry, I'm usually not use Eclipse. Have you tried to
> increase the max heap size?
>
> Run configurations -> Arguments -> VM arguments -> -Xmx1024m
>
> Or, alternatively, you can run daemons using terminals:
>
> $ bin/hama zookeeper
> $ bin/hama bspmaster
> $ bin/hama groom
>
> or
>
> $ bin/start-bspd.sh
>
> Then, you'll be able to launch a BSP job from eclipse.
>
> On Fri, Apr 25, 2014 at 6:35 PM, InJun Song <ijsong@gmail.com> wrote:
> > Thanks for your reply.
> >
> > I am running hadoop pseudo distributed mode. (apache hadoop 1.2.1)
> >
> > When I run HAMA 0.6.4 release in pseudo distributed mode (using
> > start-bspd.sh) and execute my dummy bsp job in hama commands (hama jar
> > ...), it works well.
> >
> > The problem happens when I ran daemons and my job in eclipse.
> >
> > Below is my full log of BSPMasterRunner in eclipse console.
> >
> >
> >
> > 14/04/25 18:29:01 INFO hama.BSPMasterRunner: STARTUP_MSG:
> >
> > /************************************************************
> >
> > STARTUP_MSG: Starting BSPMaster
> >
> > STARTUP_MSG:   host = km-knuth.local/172.29.9.73
> >
> > STARTUP_MSG:   args = []
> >
> > STARTUP_MSG:   version = 1.2.0
> >
> > STARTUP_MSG:   build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r
> > 1479473; compiled by 'hortonfo' on Mon May  6 18:29:07 UTC 2013
> >
> > STARTUP_MSG:   java = 1.6.0_65
> >
> > ************************************************************/
> >
> > 14/04/25 18:29:01 INFO bsp.BSPMaster: RPC BSPMaster: host localhost port
> > 40000
> >
> > 2014-04-25 18:29:01.614 java[53927:1003] Unable to load realm info from
> > SCDynamicStore
> >
> > 14/04/25 18:29:01 INFO ipc.Server: Starting Socket Reader #1 for port
> 40000
> >
> > 14/04/25 18:29:01 INFO mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> >
> > 14/04/25 18:29:01 INFO http.HttpServer: Port returned by
> > webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
> > the listener on 40013
> >
> > 14/04/25 18:29:01 INFO http.HttpServer: listener.getLocalPort() returned
> > 40013 webServer.getConnectors()[0].getLocalPort() returned 40013
> >
> > 14/04/25 18:29:01 INFO http.HttpServer: Jetty bound to port 40013
> >
> > 14/04/25 18:29:01 INFO mortbay.log: jetty-6.1.14
> >
> > 14/04/25 18:29:01 INFO mortbay.log: Started
> SelectChannelConnector@localhost
> > :40013
> >
> > 14/04/25 18:29:02 INFO bsp.BSPMaster: Cleaning up the system directory
> >
> > 14/04/25 18:29:02 INFO bsp.BSPMaster:
> > hdfs://localhost:9000/tmp/hadoop-juns/bsp/system
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52
> GMT
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client environment:host.name
> > =172.29.9.73
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:java.version=1.6.0_65
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:java.vendor=Apple Inc.
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> >
> environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> >
> environment:java.class.path=/Users/juns/workspaces/private/hama/core/target/test-classes:/Users/juns/workspaces/private/hama/core/target/classes:/Users/juns/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/juns/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/juns/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/juns/workspaces/private/hama/commons/target/test-classes:/Users/juns/workspaces/private/hama/commons/target/classes:/Users/juns/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.jar:/Users/juns/.m2/repository/org/apache/hadoop/hadoop-core/1.2.0/hadoop-core-1.2.0.jar:/Users/juns/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/juns/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/juns/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/Users/juns/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/Users/juns/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/juns/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/juns/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.2/jackson-core-asl-1.9.2.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.2/jackson-mapper-asl-1.9.2.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.7.1/jackson-jaxrs-1.7.1.jar:/Users/juns/.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar:/Users/juns/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/Users/juns/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/juns/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/Users/juns/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:/Users/juns/.m2/repository/junit/junit/4.8.1/junit-4.8.1.jar:/Users/juns/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/juns/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/juns/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/juns/.m2/repository/commons-configuration/commons-configuration/1.7/commons-configuration-1.7.jar:/Users/juns/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/juns/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/juns/.m2/repository/commons-digester/commons-digester/1.8.1/commons-digester-1.8.1.jar:/Users/juns/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar:/Users/juns/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/juns/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty/6.1.14/jetty-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty-util/6.1.14/jetty-util-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/Users/juns/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/juns/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/Users/juns/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/Users/juns/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar:/Users/juns/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/juns/.m2/repository/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar:/Users/juns/.m2/repository/hsqldb/hsqldb/
> >
> 1.8.0.10/hsqldb-1.8.0.10.jar:/Users/juns/.m2/repository/org/apache/hadoop/hadoop-test/1.2.0/hadoop-test-1.2.0.jar:/Users/juns/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/Users/juns/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/Users/juns/.m2/repository/org/slf4j/slf4j-api/1.5.8/slf4j-api-1.5.8.jar:/Users/juns/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/Users/juns/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/Users/juns/.m2/repository/commons-codec/commons-codec/1.2/commons-codec-1.2.jar:/Users/juns/.m2/repository/org/xerial/snappy/snappy-java/1.0.5/snappy-java-1.0.5.jar:/Users/juns/.m2/repository/org/apache/commons/commons-math3/3.0/commons-math3-3.0.jar:/Users/juns/.m2/repository/org/apache/ant/ant/1.7.1/ant-1.7.1.jar:/Users/juns/.m2/repository/org/apache/ant/ant-launcher/1.7.1/ant-launcher-1.7.1.jar:/Users/juns/.m2/repository/org/apache/tomcat/servlet-api/6.0.32/servlet-api-6.0.32.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty-annotations/6.1.14/jetty-annotations-6.1.14.jar:/Users/juns/.m2/repository/org/mortbay/jetty/jetty-plus/6.1.14/jetty-plus-6.1.14.jar:/Users/juns/.m2/repository/geronimo-spec/geronimo-spec-jta/1.0.1B-rc4/geronimo-spec-jta-1.0.1B-rc4.jar:/Users/juns/.m2/repository/org/apache/geronimo/specs/geronimo-annotation_1.0_spec/1.0/geronimo-annotation_1.0_spec-1.0.jar:/Users/juns/.m2/repository/org/slf4j/slf4j-log4j12/1.5.8/slf4j-log4j12-1.5.8.jar:/Users/juns/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/Users/juns/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/juns/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/juns/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar:/Users/juns/workspaces/private/hama/conf
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> >
> environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> >
> environment:java.io.tmpdir=/var/folders/rv/42slk6553x7885qkq08kdt3r0000gn/T/
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:java.compiler=<NA>
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Mac
> > OS X
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:os.arch=x86_64
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:os.version=10.9.2
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client environment:user.name
> > =juns
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:user.home=/Users/juns
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Client
> > environment:user.dir=/Users/juns/workspaces/private/hama/core
> >
> > 14/04/25 18:29:02 INFO zookeeper.ZooKeeper: Initiating client connection,
> > connectString=localhost:21810 sessionTimeout=1200000
> > watcher=org.apache.hama.bsp.sync.ZKSyncBSPMasterClient@76d78df0
> >
> > 14/04/25 18:29:02 INFO sync.ZKSyncBSPMasterClient: Initialized ZK false
> >
> > 14/04/25 18:29:02 INFO sync.ZKSyncClient: Initializing ZK Sync Client
> >
> > 14/04/25 18:29:02 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server localhost/0:0:0:0:0:0:0:1:21810. Will not attempt to authenticate
> > using SASL (Unable to locate a login configuration)
> >
> > 14/04/25 18:29:02 INFO zookeeper.ClientCnxn: Socket connection
> established
> > to localhost/0:0:0:0:0:0:0:1:21810, initiating session
> >
> > 14/04/25 18:29:02 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server localhost/0:0:0:0:0:0:0:1:21810, sessionid = 0x145983756eb0000,
> > negotiated timeout = 1200000
> >
> > 14/04/25 18:29:02 INFO ipc.Server: IPC Server Responder: starting
> >
> > 14/04/25 18:29:02 INFO ipc.Server: IPC Server listener on 40000: starting
> >
> > 14/04/25 18:29:02 INFO ipc.Server: IPC Server handler 0 on 40000:
> starting
> >
> > 14/04/25 18:29:02 INFO bsp.BSPMaster: Starting RUNNING
> >
> > 14/04/25 18:29:05 INFO bsp.BSPMaster: groomd_172.29.9.73_50000 is added.
> >
> > Exception in thread "Socket Reader #1 for port 40000"
> > java.lang.OutOfMemoryError: Java heap space
> >
> > at org.apache.hama.ipc.RPC$Invocation.readFields(RPC.java:111)
> >
> > at org.apache.hama.ipc.Server$Connection.processData(Server.java:1121)
> >
> > at org.apache.hama.ipc.Server$Connection.processOneRpc(Server.java:1099)
> >
> > at org.apache.hama.ipc.Server$Connection.readAndProcess(Server.java:1063)
> >
> > at org.apache.hama.ipc.Server$Listener.doRead(Server.java:576)
> >
> >  at org.apache.hama.ipc.Server$Listener$Reader.run(Server.java:363)
> >
> >
> > On Fri, Apr 25, 2014 at 4:57 PM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >
> >> Hi, can you attach the full log of the BSPMaster?
> >>
> >> Just to be sure, to run Hama in pseudo distributed mode, you must run
> >> HDFS first.
> >>
> >> On Fri, Apr 25, 2014 at 4:19 PM, InJun Song <ijsong@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I have a question about how to run hama in eclipse environment.
> >> >
> >> > At first, I set up my develop environment according to this page. (
> >> > http://wiki.apache.org/hama/HowToContribute)
> >> >
> >> > Then, because I wanted to run pseudo distributed mode, I ran each
> daemon
> >> > under eclipse. (zookeeper -> bspmaster -> groomserver). To run those,
> I
> >> > executed three runner for them; ZookeeprRunner, BSPMasterRunner and
> >> > GroomserverRunner.
> >> > After all daemons launched, I ran dummy BSP program.
> >> >
> >> > At that time, BSPMaster throws OOM exception. Code throws an
> exception is
> >> > in "RPC.Invocation.readFields" method. While it executed this line, it
> >> > throws the exception.
> >> >
> >> >     parameters = new Object[in.readInt()];
> >> >
> >> > The number of object to create is 308766068. If
> sizeof(java.lang.Object)
> >> is
> >> > 8 byte, total size is over 2G. I think that it must be wrong.
> >> >
> >> > What did I mistake? What mistake make me go wrong way?
> >> > I am debugging with eclipse, but I want to some advices from you.
> >> >
> >> > Thank you.
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> CEO at DataSayer Co., Ltd.
> >>
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message