Return-Path: X-Original-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 437E29663 for ; Fri, 21 Oct 2011 16:36:57 +0000 (UTC) Received: (qmail 41705 invoked by uid 500); 21 Oct 2011 16:36:57 -0000 Delivered-To: apmail-incubator-hama-dev-archive@incubator.apache.org Received: (qmail 41683 invoked by uid 500); 21 Oct 2011 16:36:57 -0000 Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-dev@incubator.apache.org Received: (qmail 41675 invoked by uid 99); 21 Oct 2011 16:36:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2011 16:36:57 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2011 16:36:53 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 735A3315E4A for ; Fri, 21 Oct 2011 16:34:32 +0000 (UTC) Date: Fri, 21 Oct 2011 16:34:32 +0000 (UTC) From: "Thomas Jungblut (Issue Comment Edited) (JIRA)" To: hama-dev@incubator.apache.org Message-ID: <120646855.1739.1319214872474.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <403292122.19660.1318829411817.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Issue Comment Edited] (HAMA-454) Add Zookeeper as synchronization service MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HAMA-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132803#comment-13132803 ] Thomas Jungblut edited comment on HAMA-454 at 10/21/11 4:34 PM: ---------------------------------------------------------------- Zookeeper now works (theoretically). Sometimes I still have shutdown issues in the ApplicationMaster: {noformat} 11/10/21 18:30:57 INFO server.NIOServerCnxn: NIOServerCnxn factory exited run method 11/10/21 18:30:57 INFO server.PrepRequestProcessor: PrepRequestProcessor exited loop! 11/10/21 18:30:57 INFO server.SyncRequestProcessor: SyncRequestProcessor exited! 11/10/21 18:30:57 INFO server.FinalRequestProcessor: shutdown of request processor complete 11/10/21 18:30:57 INFO server.FinalRequestProcessor: shutdown of request processor complete 11/10/21 18:30:57 WARN jmx.MBeanRegistry: Failed to unregister MBean InMemoryDataTree 11/10/21 18:30:57 WARN jmx.MBeanRegistry: Error during unregister javax.management.InstanceNotFoundException: org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403) at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:94) at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:111) at org.apache.zookeeper.server.ZooKeeperServer.unregisterJMX(ZooKeeperServer.java:421) at org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:414) at org.apache.zookeeper.server.NIOServerCnxn$Factory.shutdown(NIOServerCnxn.java:323) at org.apache.zookeeper.server.ZooKeeperServerMain.shutdown(ZooKeeperServerMain.java:125) at org.apache.hama.zookeeper.QuorumPeer$ShutdownableZooKeeperServerMain.shutdown(QuorumPeer.java:407) at org.apache.hama.zookeeper.QuorumPeer$ShutdownableZooKeeperServerMain.shutdownZookeeperMain(QuorumPeer.java:402) at org.apache.hama.bsp.sync.zookeeper.ZooKeeperSyncServerImpl.stopServer(ZooKeeperSyncServerImpl.java:82) at org.apache.hama.bsp.sync.SyncServerRunner.stop(SyncServerRunner.java:47) at org.apache.hama.bsp.BSPApplicationMaster.cleanup(BSPApplicationMaster.java:243) at org.apache.hama.bsp.BSPApplicationMaster.main(BSPApplicationMaster.java:279) {noformat} Besides this everything works. Even the RPC Synchronization works, but we have to manually build core module with Hadoop 23.0. Unit tests complete as well: {noformat} [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hama parent POM ............................ SUCCESS [3.121s] [INFO] Apache Hama Core .................................. SUCCESS [11.107s] [INFO] Apache Hama Graph Package ......................... SUCCESS [2.572s] [INFO] Apache Hama Examples .............................. SUCCESS [25.469s] [INFO] Apache Hama YARN .................................. SUCCESS [46.075s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:30.192s [INFO] Finished at: Fri Oct 21 18:29:10 CEST 2011 [INFO] Final Memory: 35M/353M [INFO] ------------------------------------------------------------------------ {noformat} was (Author: thomas.jungblut): Zookeeper now works (theoretically). Sometimes I still have shutdown issues in the ApplicationMaster: {noformat} 11/10/21 18:30:57 INFO server.NIOServerCnxn: NIOServerCnxn factory exited run method 11/10/21 18:30:57 INFO server.PrepRequestProcessor: PrepRequestProcessor exited loop! 11/10/21 18:30:57 INFO server.SyncRequestProcessor: SyncRequestProcessor exited! 11/10/21 18:30:57 INFO server.FinalRequestProcessor: shutdown of request processor complete 11/10/21 18:30:57 INFO server.FinalRequestProcessor: shutdown of request processor complete 11/10/21 18:30:57 WARN jmx.MBeanRegistry: Failed to unregister MBean InMemoryDataTree 11/10/21 18:30:57 WARN jmx.MBeanRegistry: Error during unregister javax.management.InstanceNotFoundException: org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403) at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:94) at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:111) at org.apache.zookeeper.server.ZooKeeperServer.unregisterJMX(ZooKeeperServer.java:421) at org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:414) at org.apache.zookeeper.server.NIOServerCnxn$Factory.shutdown(NIOServerCnxn.java:323) at org.apache.zookeeper.server.ZooKeeperServerMain.shutdown(ZooKeeperServerMain.java:125) at org.apache.hama.zookeeper.QuorumPeer$ShutdownableZooKeeperServerMain.shutdown(QuorumPeer.java:407) at org.apache.hama.zookeeper.QuorumPeer$ShutdownableZooKeeperServerMain.shutdownZookeeperMain(QuorumPeer.java:402) at org.apache.hama.bsp.sync.zookeeper.ZooKeeperSyncServerImpl.stopServer(ZooKeeperSyncServerImpl.java:82) at org.apache.hama.bsp.sync.SyncServerRunner.stop(SyncServerRunner.java:47) at org.apache.hama.bsp.BSPApplicationMaster.cleanup(BSPApplicationMaster.java:243) at org.apache.hama.bsp.BSPApplicationMaster.main(BSPApplicationMaster.java:279) {noformat} Besides this everything works. Even the RPC Synchronization works, but we have to manually build core module with Hadoop 23.0. > Add Zookeeper as synchronization service > ---------------------------------------- > > Key: HAMA-454 > URL: https://issues.apache.org/jira/browse/HAMA-454 > Project: Hama > Issue Type: Sub-task > Reporter: Thomas Jungblut > Assignee: Thomas Jungblut > Attachments: HAMA-454.patch, HAMA-454.patch, HAMA-454_v2.patch > > > We should use Zookeeper instead of our own implementation. > Additionally we should use the plain BSPPeerImpl in YARN to reduce duplicate code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira