Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 85FA71162F for ; Fri, 1 Aug 2014 01:08:50 +0000 (UTC) Received: (qmail 51787 invoked by uid 500); 1 Aug 2014 01:08:45 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 51654 invoked by uid 500); 1 Aug 2014 01:08:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 51642 invoked by uid 99); 1 Aug 2014 01:08:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Aug 2014 01:08:45 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.216.45 is neither permitted nor denied by domain of discord@uw.edu) Received: from [209.85.216.45] (HELO mail-qa0-f45.google.com) (209.85.216.45) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Aug 2014 01:08:41 +0000 Received: by mail-qa0-f45.google.com with SMTP id cm18so3188862qab.18 for ; Thu, 31 Jul 2014 18:08:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=6z12xuok4++EvxN58DEMDZ9bIidGqUOxoHSOuyBJ4Fk=; b=Sey/9ix/6lIDEff9m0RrnRKKFVdUfPDoBlxdvD9KtneCwPo8YhcKNB05PC7ypB3RYW xNqU9DdQNRNAU4e7u5Hul4fuKwCvuB2bUsgCRwmRydS7SgusJq0d3OUtn/ZIveTY7lak zd5viZMRx0eQqtF1pyO5cL0vNxN0ymLQI4GzSrKbP6ZD8P8wzQY55ESIFY+KcqNgJQlK 5XNiZoA2s66K4g5FmHdhpxFpuJM7BkixYNIKGIhPAAd+iao6xWYF3+MS2uhh1/im6Pgz Ms9zNyJI0vYbOVkFqm2AdNuwEUfNy8HZowlGan1PuuP1o/Zk3aOswc8Csh9tkArCbE7g XPMQ== X-Gm-Message-State: ALoCoQnoxRzMm6Kq2S/470eCC8FQ4I41Li0G5vIL0ircEei1pkwimct7TUSzQRvh8R4Y3x/+kZUy MIME-Version: 1.0 X-Received: by 10.224.88.71 with SMTP id z7mr3351245qal.94.1406855300291; Thu, 31 Jul 2014 18:08:20 -0700 (PDT) Received: by 10.140.82.38 with HTTP; Thu, 31 Jul 2014 18:08:20 -0700 (PDT) In-Reply-To: References: Date: Thu, 31 Jul 2014 18:08:20 -0700 Message-ID: Subject: Re: Juggling or swaping out the standby NameNode in a QJM / HA configuration From: Colin Kincaid Williams To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1133da306cc56604ff87069d X-Virus-Checked: Checked by ClamAV on apache.org --001a1133da306cc56604ff87069d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I tried a third time and it just worked? sudo hdfs zkfc -formatZK 2014-07-31 18:07:51,595 INFO [main] tools.DFSZKFailoverController (DFSZKFailoverController.java:(140)) - Failover controller configured for NameNode NameNode at rhel1.local/10.120.5.203:8020 2014-07-31 18:07:51,791 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:zookeeper.version=3D3.4.3-cdh4.1.3--1, built on 01/27/2013 00:1= 3 GMT 2014-07-31 18:07:51,791 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:host.name=3Drhel1.local 2014-07-31 18:07:51,792 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.version=3D1.7.0_60 2014-07-31 18:07:51,792 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.vendor=3DOracle Corporation 2014-07-31 18:07:51,792 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.home=3D/usr/java/jdk1.7.0_60/jre 2014-07-31 18:07:51,792 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.class.path=3D/etc/hadoop/conf:/usr/lib/hadoop/lib/jersey-c= ore-1.8.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jax= b-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/li= b/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoo= p/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/= commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/had= oop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/us= r/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-collections-3.= 2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/= guava-11.0.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/= hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/li= b/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jersey-server-1.= 8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/ja= ckson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/= lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hado= op/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.ja= r:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-runti= me-5.5.23.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/= lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.ja= r:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/activa= tion-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/= hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5= .jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/junit-= 4.8.2.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/common= s-math-2.1.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/li= b/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/l= ib/paranamer-2.3.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/= hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:= /usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/.//hadoop-annotat= ions-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoo= p/.//hadoop-common-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop/.//hadoop-commo= n-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/had= oop/.//hadoop-auth-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/u= sr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib= /hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mappe= r-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/l= ib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr= /lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.= cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdf= s/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:= /usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/= lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/= hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94= .jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/c= ommons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/li= b/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-= util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/u= sr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/co= mmons-codec-1.4.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3.jar:= /usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoo= p-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr= /lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/= asm-3.2.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-= yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1= .8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/a= vro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/us= r/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1= .jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/= javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/li= b/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/paranamer-= 2.3.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-y= arn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-api-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-si= te-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4= .1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanag= er.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop= -yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applica= tions-distributedshell-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-ya= rn-server-web-proxy-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-= server-tests-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-s= erver-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-pr= oxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cd= h4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.1.3= .jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.1.3.= jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-0.20-map= reduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/h= adoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6= .5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-m= apreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapred= uce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging= -1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/had= oop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/j= etty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.j= ar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20= -mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-= api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapr= educe/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-c= ompiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.j= ar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.= 20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/comm= ons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1= .6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce= /lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-d= igester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-map= reduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.= jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/us= r/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop= -0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib= /avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snap= py-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapr= educe/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-a= ll-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/h= adoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/= lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.ja= r:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hado= op-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/li= b/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairsched= uler-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-exampl= es-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr= 1-cdh4.1.3-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant-2.0.0-mr1-c= dh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoo= p-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-core.jar:/usr/lib/hadoop-0.20= -mapreduce/.//hadoop-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tool= s.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core-2.0.0-mr1-cdh4.1.3.jar:= /usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-examples.jar:/u= sr/lib/hadoop-0.20-mapreduce/.//hadoop-test-2.0.0-mr1-cdh4.1.3.jar:/usr/lib= /hadoop-0.20-mapreduce/.//hadoop-tools-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hado= op-0.20-mapreduce/.//hadoop-examples.jar:/usr/lib/hadoop-0.20-mapreduce/.//= hadoop-2.0.0-mr1-cdh4.1.3-tools.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoo= p-2.0.0-mr1-cdh4.1.3-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core= .jar 2014-07-31 18:07:51,793 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.library.path=3D//usr/lib/hadoop/lib/native 2014-07-31 18:07:51,801 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=3D/tmp 2014-07-31 18:07:51,801 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler=3D 2014-07-31 18:07:51,801 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=3DLinux 2014-07-31 18:07:51,802 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=3Damd64 2014-07-31 18:07:51,802 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=3D2.6.32-358.el6.x86_64 2014-07-31 18:07:51,802 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=3Droot 2014-07-31 18:07:51,802 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=3D/root 2014-07-31 18:07:51,802 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=3D/etc/hbase/conf.golden_apple 2014-07-31 18:07:51,813 INFO [main] zookeeper.ZooKeeper (ZooKeeper.java:(433)) - Initiating client connection, connectString=3Drhel1.local:2181,rhel6.local:2181,rhel2.local:2181 sessionTimeout=3D5000 watcher=3Dnull 2014-07-31 18:07:51,833 INFO [main-SendThread(rhel1.local:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(958)) - Opening socket connection to server rhel1.local/10.120.5.203:2181. Will not attempt to authenticate using SASL (unknown error) 2014-07-31 18:07:51,844 INFO [main-SendThread(rhel1.local:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(850)) - Socket connection established to rhel1.local/10.120.5.203:2181, initiating session 2014-07-31 18:07:51,852 INFO [main-SendThread(rhel1.local:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1187)) - Session establishment complete on server rhel1.local/10.120.5.203:2181, sessionid = =3D 0x1478902fddc000a, negotiated timeout =3D 5000 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The configured parent znode /hadoop-ha/golden-apple already exists. Are you sure you want to clear all failover information from ZooKeeper? WARNING: Before proceeding, ensure that all HDFS services and failover controllers are stopped! =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Proceed formatting /hadoop-ha/golden-apple? (Y or N) 2014-07-31 18:07:51,858 INFO [main-EventThread] ha.ActiveStandbyElector (ActiveStandbyElector.java:processWatchEvent(538)) - Session connected. Y 2014-07-31 18:08:00,439 INFO [main] ha.ActiveStandbyElector (ActiveStandbyElector.java:clearParentZNode(314)) - Recursively deleting /hadoop-ha/golden-apple from ZK... 2014-07-31 18:08:00,506 INFO [main] ha.ActiveStandbyElector (ActiveStandbyElector.java:clearParentZNode(327)) - Successfully deleted /hadoop-ha/golden-apple from ZK. 2014-07-31 18:08:00,541 INFO [main] ha.ActiveStandbyElector (ActiveStandbyElector.java:ensureParentZNode(299)) - Successfully created /hadoop-ha/golden-apple in ZK. 2014-07-31 18:08:00,545 INFO [main-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(511)) - EventThread shut down 2014-07-31 18:08:00,545 INFO [main] zookeeper.ZooKeeper (ZooKeeper.java:close(679)) - Session: 0x1478902fddc000a closed On Thu, Jul 31, 2014 at 2:51 PM, Alex Newman wrote: > Cheers. That's rough. We don't have that problem here at WanDISCO. > > On Thu, Jul 31, 2014 at 12:46 PM, Colin Kincaid Williams > wrote: > > Hi this is drocsid / discord from #hbase. Thanks for the help earlier > today. > > Just thought I'd forward this info regarding swapping out the NameNode > in a > > QJM / HA configuration. See you around on #hbase. If you visit Seattle, > feel > > free to give me a shout out. > > > > ---------- Forwarded message ---------- > > From: Colin Kincaid Williams > > Date: Thu, Jul 31, 2014 at 12:35 PM > > Subject: Re: Juggling or swaping out the standby NameNode in a QJM / HA > > configuration > > To: user@hadoop.apache.org > > > > > > Hi Jing, > > > > Thanks for the response. I will try this out, and file an Apache jira. > > > > Best, > > > > Colin Williams > > > > > > On Thu, Jul 31, 2014 at 11:19 AM, Jing Zhao > wrote: > >> > >> Hi Colin, > >> > >> I guess currently we may have to restart almost all the > >> daemons/services in order to swap out a standby NameNode (SBN): > >> > >> 1. The current active NameNode (ANN) needs to know the new SBN since i= n > >> the current implementation the SBN tries to send rollEditLog RPC > request to > >> ANN periodically (thus if a NN failover happens later, the original AN= N > >> needs to send this RPC to the correct NN). > >> 2. Looks like the DataNode currently cannot do real refreshment for NN= . > >> Look at the code in BPOfferService: > >> > >> void refreshNNList(ArrayList addrs) throws > >> IOException { > >> Set oldAddrs =3D Sets.newHashSet(); > >> for (BPServiceActor actor : bpServices) { > >> oldAddrs.add(actor.getNNSocketAddress()); > >> } > >> Set newAddrs =3D Sets.newHashSet(addrs); > >> > >> if (!Sets.symmetricDifference(oldAddrs, newAddrs).isEmpty()) { > >> // Keep things simple for now -- we can implement this at a late= r > >> date. > >> throw new IOException( > >> "HA does not currently support adding a new standby to a > running > >> DN. " + > >> "Please do a rolling restart of DNs to reconfigure the list = of > >> NNs."); > >> } > >> } > >> > >> 3. If you're using automatic failover, you also need to update the > >> configuration of the ZKFC on the current ANN machine, since ZKFC will = do > >> gracefully fencing by sending RPC to the other NN. > >> 4. Looks like we do not need to restart JournalNodes for the new SBN > but I > >> have not tried before. > >> > >> Thus in general we may still have to restart all the services > (except > >> JNs) and update their configurations. But this may be a rolling restar= t > >> process I guess: > >> 1. Shutdown the old SBN, bootstrap the new SBN, and start the new SBN. > >> 2. Keep the ANN and its corresponding ZKFC running, do a rolling resta= rt > >> of all the DN to update their configurations > >> 3. After restarting all the DN, stop ANN and the ZKFC, and update thei= r > >> configuration. The new SBN should become active. > >> > >> I have not tried the upper steps, thus please let me know if this > >> works or not. And I think we should also document the correct steps in > >> Apache. Could you please file an Apache jira? > >> > >> Thanks, > >> -Jing > >> > >> > >> > >> On Thu, Jul 31, 2014 at 9:37 AM, Colin Kincaid Williams > > >> wrote: > >>> > >>> Hello, > >>> > >>> I'm trying to swap out a standby NameNode in a QJM / HA configuration= . > I > >>> believe the steps to achieve this would be something similar to: > >>> > >>> Use the Bootstrap standby command to prep the replacment standby. Or > >>> rsync if the command fails. > >>> > >>> Somehow update the datanodes, so they push the heartbeat / journal to > the > >>> new standby > >>> > >>> Update the xml configuration on all nodes to reflect the replacment > >>> standby. > >>> > >>> Start the replacment standby > >>> > >>> Use some hadoop command to refresh the datanodes to the new NameNode > >>> configuration. > >>> > >>> I am not sure how to deal with the Journal switch, or if I am going > about > >>> this the right way. Can anybody give me some suggestions here? > >>> > >>> > >>> Regards, > >>> > >>> Colin Williams > >>> > >> > >> > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or enti= ty > >> to which it is addressed and may contain information that is > confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader of > >> this message is not the intended recipient, you are hereby notified > that any > >> printing, copying, dissemination, distribution, disclosure or > forwarding of > >> this communication is strictly prohibited. If you have received this > >> communication in error, please contact the sender immediately and > delete it > >> from your system. Thank You. > > > > > > > --001a1133da306cc56604ff87069d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I tried a third time and it just worked?

sudo hdfs zkfc -formatZK
2014-07-31 18:07:51,595 INFO =C2= =A0[main] tools.DFSZKFailoverController (DFSZKFailoverController.java:<i= nit>(140)) - Failover controller configured for NameNode NameNode at rhe= l1.local/10.120.5.203:8020
2014-07-31 18:07:51,791 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:zookeeper.version=3D3.4.3-cdh4.= 1.3--1, built on 01/27/2013 00:13 GMT
2014-07-31 18:07:51,791 INF= O =C2=A0[main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client = environment:host.name=3Drhel1.local
2014-07-31 18:07:51,792 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.version=3D1.7.0_60
2014-07-31 18:07:51,792 INFO =C2=A0[main] zookeeper.ZooKeeper (Environme= nt.java:logEnv(100)) - Client environment:java.vendor=3DOracle Corporation<= /div>
2014-07-31 18:07:51,792 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.home=3D/usr/java/jdk1.7.0_= 60/jre
2014-07-31 18:07:51,792 INFO =C2=A0[main] zookeeper.ZooKee= per (Environment.java:logEnv(100)) - Client environment:java.class.path=3D/= etc/hadoop/conf:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib= /commons-cli-1.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/had= oop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/u= sr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hado= op/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/= lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/li= b/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jsp-api-2.1.ja= r:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/jac= kson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoo= p/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:= /usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/avro-1.7.1.clouder= a.2.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jaspe= r-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/= hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib= /hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.= 6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jackson-x= c-1.8.8.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/l= ib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/li= b/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-h= ttpclient-3.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/li= b/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.= 7.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/snappy= -java-1.0.4.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/s= tax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/= lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/had= oop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/h= adoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:= /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/commons-cod= ec-1.4.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.1.3.jar:/usr/li= b/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.1= .3-tests.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.1.3.jar:/usr/lib/h= adoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.1= .3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/= hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.= 2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop= -hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/ha= doop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.5= 2.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-= hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.ja= r:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/zookee= per-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/li= b/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1= .8.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/c= ommons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hado= op-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtim= e-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr= /lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf= -java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/ha= doop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/.//hadoop-= hdfs-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/= lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/jackson-ma= pper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yar= n/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8= .jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib= /log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/avro-1.7.1.cloudera.2.jar:/usr/l= ib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0= .jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/a= opalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/had= oop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-yarn/lib/guice-servlet= -3.0.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/li= b/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications= -distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.1.= 3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.1.3.jar:/usr/lib/= hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib= /hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//= hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tes= ts-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar= :/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cd= h4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.= 1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.1.3-tes= ts.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/= lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/./= /hadoop-yarn-server-resourcemanager-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn= /.//hadoop-yarn-server-common-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//ha= doop-yarn-server-nodemanager-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//had= oop-yarn-common.jar:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-= mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-co= ntrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-map= reduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson= -mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/u= sr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce= /lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/l= ib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20= -mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0= .52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jac= kson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.2= 0-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.= 7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/had= oop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/= lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-m= apreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson= -xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:= /usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop= -0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/= commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons= -httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/l= ib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapred= uce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib= /commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lan= g-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/= stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:= /usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.= 20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-a= pi-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduc= e/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-js= on-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.3.jar:/us= r/lib/hadoop-0.20-mapreduce/.//hadoop-examples-2.0.0-mr1-cdh4.1.3.jar:/usr/= lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-ant.jar:/usr/lib/had= oop-0.20-mapreduce/.//hadoop-ant-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.2= 0-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.= 0-mr1-cdh4.1.3-core.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test.jar:/= usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools.jar:/usr/lib/hadoop-0.20-mapr= educe/.//hadoop-core-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/= .//hadoop-2.0.0-mr1-cdh4.1.3-examples.jar:/usr/lib/hadoop-0.20-mapreduce/./= /hadoop-test-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoo= p-tools-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-exa= mples.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-tools= .jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-test.jar:/= usr/lib/hadoop-0.20-mapreduce/.//hadoop-core.jar
2014-07-31 18:07:51,793 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.library.path=3D//usr/lib/h= adoop/lib/native
2014-07-31 18:07:51,801 INFO =C2=A0[main] zookee= per.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.t= mpdir=3D/tmp
2014-07-31 18:07:51,801 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.compiler=3D<NA>
2014-07-31 18:07:51,801 INFO =C2=A0[main] zookeeper.ZooKeeper (Enviro= nment.java:logEnv(100)) - Client environment:os.= name=3DLinux
2014-07-31 18:07:51,802 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:os.arch=3Damd64
2014-= 07-31 18:07:51,802 INFO =C2=A0[main] zookeeper.ZooKeeper (Environment.java:= logEnv(100)) - Client environment:os.version=3D2.6.32-358.el6.x86_64
2014-07-31 18:07:51,802 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:us= er.name=3Droot
2014-07-31 18:07:51,802 INFO =C2=A0[main] zook= eeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.ho= me=3D/root
2014-07-31 18:07:51,802 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:user.dir=3D/etc/hbase/conf.gold= en_apple
2014-07-31 18:07:51,813 INFO =C2=A0[main] zookeeper.ZooK= eeper (ZooKeeper.java:<init>(433)) - Initiating client connection, co= nnectString=3Drhel1.local:2181,rhel6.local:2181,rhel2.local:2181 sessionTim= eout=3D5000 watcher=3Dnull
2014-07-31 18:07:51,833 INFO =C2=A0[main-SendThread(rhel1.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(958)) - Opening sock= et connection to server rhel1.local/10= .120.5.203:2181. Will not attempt to authenticate using SASL (unknown e= rror)
2014-07-31 18:07:51,844 INFO =C2=A0[main-SendThread(rhel1.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(850)) - Socket conne= ction established to rhel1.local/10.12= 0.5.203:2181, initiating session
2014-07-31 18:07:51,852 INFO =C2=A0[main-SendThread(rhel1.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1187)) - Session establi= shment complete on server rhel1.local/= 10.120.5.203:2181, sessionid =3D 0x1478902fddc000a, negotiated timeout = =3D 5000
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
The configured parent znode /hadoop-ha/golden-apple already exists.=
Are you sure you want to clear all failover information from
ZooKeeper?
WARNING: Before proceeding, ensure that all HDFS services and
failover controllers are stopped!
=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Proceed formatting /hadoo= p-ha/golden-apple? (Y or N) 2014-07-31 18:07:51,858 INFO =C2=A0[main-EventT= hread] ha.ActiveStandbyElector (ActiveStandbyElector.java:processWatchEvent= (538)) - Session connected.
Y=C2=A0
2014-07-31 18:08:00,439 INFO =C2=A0[main] ha.ActiveS= tandbyElector (ActiveStandbyElector.java:clearParentZNode(314)) - Recursive= ly deleting /hadoop-ha/golden-apple from ZK...
2014-07-31 18:08:0= 0,506 INFO =C2=A0[main] ha.ActiveStandbyElector (ActiveStandbyElector.java:= clearParentZNode(327)) - Successfully deleted /hadoop-ha/golden-apple from = ZK.
2014-07-31 18:08:00,541 INFO =C2=A0[main] ha.ActiveStandbyElector (Act= iveStandbyElector.java:ensureParentZNode(299)) - Successfully created /hado= op-ha/golden-apple in ZK.
2014-07-31 18:08:00,545 INFO =C2=A0[mai= n-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(511)) - EventThrea= d shut down
2014-07-31 18:08:00,545 INFO =C2=A0[main] zookeeper.ZooKeeper (ZooKeep= er.java:close(679)) - Session: 0x1478902fddc000a closed



On Thu, Jul = 31, 2014 at 2:51 PM, Alex Newman <posix4e@gmail.com> wrote:<= br>
Cheers. That's rough. We don't have that problem h= ere at WanDISCO.

On Thu, Jul 31, 2014 at 12:46 PM, Colin Kincaid Williams <discord@uw.edu> wrote:
> Hi this is drocsid / discord from #hbase. Thanks for the help earlier = today.
> Just thought I'd forward this info regarding swapping out the Name= Node in a
> QJM / HA configuration. See you around on #hbase. If you visit Seattle= , feel
> free to give me a shout out.
>
> ---------- Forwarded message ----------
> From: Colin Kincaid Williams <discord@uw.edu>
> Date: Thu, Jul 31, 2014 at 12:35 PM
> Subject: Re: Juggling or swaping out the standby NameNode in a QJM / H= A
> configuration
> To: user@h= adoop.apache.org
>
>
> Hi Jing,
>
> Thanks for the response. I will try this out, and file an Apache jira.=
>
> Best,
>
> Colin Williams
>
>
> On Thu, Jul 31, 2014 at 11:19 AM, Jing Zhao <jing@hortonworks.com> wrote:
>>
>> Hi Colin,
>>
>> =C2=A0 =C2=A0 I guess currently we may have to restart almost all = the
>> daemons/services in order to swap out a standby NameNode (SBN): >>
>> 1. The current active NameNode (ANN) needs to know the new SBN sin= ce in
>> the current implementation the SBN tries to send rollEditLog RPC r= equest to
>> ANN periodically (thus if a NN failover happens later, the origina= l ANN
>> needs to send this RPC to the correct NN).
>> 2. Looks like the DataNode currently cannot do real refreshment fo= r NN.
>> Look at the code in BPOfferService:
>>
>> =C2=A0 void refreshNNList(ArrayList<InetSocketAddress> addrs= ) throws
>> IOException {
>> =C2=A0 =C2=A0 Set<InetSocketAddress> oldAddrs =3D Sets.newHa= shSet();
>> =C2=A0 =C2=A0 for (BPServiceActor actor : bpServices) {
>> =C2=A0 =C2=A0 =C2=A0 oldAddrs.add(actor.getNNSocketAddress());
>> =C2=A0 =C2=A0 }
>> =C2=A0 =C2=A0 Set<InetSocketAddress> newAddrs =3D Sets.newHa= shSet(addrs);
>>
>> =C2=A0 =C2=A0 if (!Sets.symmetricDifference(oldAddrs, newAddrs).is= Empty()) {
>> =C2=A0 =C2=A0 =C2=A0 // Keep things simple for now -- we can imple= ment this at a later
>> date.
>> =C2=A0 =C2=A0 =C2=A0 throw new IOException(
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "HA does not currently sup= port adding a new standby to a running
>> DN. " +
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "Please do a rolling resta= rt of DNs to reconfigure the list of
>> NNs.");
>> =C2=A0 =C2=A0 }
>> =C2=A0 }
>>
>> 3. If you're using automatic failover, you also need to update= the
>> configuration of the ZKFC on the current ANN machine, since ZKFC w= ill do
>> gracefully fencing by sending RPC to the other NN.
>> 4. Looks like we do not need to restart JournalNodes for the new S= BN but I
>> have not tried before.
>>
>> =C2=A0 =C2=A0 Thus in general we may still have to restart all the= services (except
>> JNs) and update their configurations. But this may be a rolling re= start
>> process I guess:
>> 1. Shutdown the old SBN, bootstrap the new SBN, and start the new = SBN.
>> 2. Keep the ANN and its corresponding ZKFC running, do a rolling r= estart
>> of all the DN to update their configurations
>> 3. After restarting all the DN, stop ANN and the ZKFC, and update = their
>> configuration. The new SBN should become active.
>>
>> =C2=A0 =C2=A0 I have not tried the upper steps, thus please let me= know if this
>> works or not. And I think we should also document the correct step= s in
>> Apache. Could you please file an Apache jira?
>>
>> Thanks,
>> -Jing
>>
>>
>>
>> On Thu, Jul 31, 2014 at 9:37 AM, Colin Kincaid Williams <discord@uw.edu>
>> wrote:
>>>
>>> Hello,
>>>
>>> I'm trying to swap out a standby NameNode in a QJM / HA co= nfiguration. I
>>> believe the steps to achieve this would be something similar t= o:
>>>
>>> Use the Bootstrap standby command to prep the replacment stand= by. Or
>>> rsync if the command fails.
>>>
>>> Somehow update the datanodes, so they push the heartbeat / jou= rnal to the
>>> new standby
>>>
>>> Update the xml configuration on all nodes to reflect the repla= cment
>>> standby.
>>>
>>> Start the replacment standby
>>>
>>> Use some hadoop command to refresh the datanodes to the new Na= meNode
>>> configuration.
>>>
>>> I am not sure how to deal with the Journal switch, or if I am = going about
>>> this the right way. Can anybody give me some suggestions here?=
>>>
>>>
>>> Regards,
>>>
>>> Colin Williams
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or = entity
>> to which it is addressed and may contain information that is confi= dential,
>> privileged and exempt from disclosure under applicable law. If the= reader of
>> this message is not the intended recipient, you are hereby notifie= d that any
>> printing, copying, dissemination, distribution, disclosure or forw= arding of
>> this communication is strictly prohibited. If you have received th= is
>> communication in error, please contact the sender immediately and = delete it
>> from your system. Thank You.
>
>
>

--001a1133da306cc56604ff87069d--