Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 984A4115F5 for ; Fri, 1 Aug 2014 01:00:03 +0000 (UTC) Received: (qmail 33394 invoked by uid 500); 1 Aug 2014 00:59:57 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 33298 invoked by uid 500); 1 Aug 2014 00:59:57 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 33288 invoked by uid 99); 1 Aug 2014 00:59:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Aug 2014 00:59:56 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 209.85.192.49 is neither permitted nor denied by domain of discord@uw.edu) Received: from [209.85.192.49] (HELO mail-qg0-f49.google.com) (209.85.192.49) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Aug 2014 00:59:49 +0000 Received: by mail-qg0-f49.google.com with SMTP id j107so4997282qga.8 for ; Thu, 31 Jul 2014 17:59:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=czojrnUVBL6HTUEoWAx58xV6XmcOPT27c0uNwbvu4JA=; b=Rva5solr6k++lcVbPEIvxZaErMsYiNEu2eIwFi1umaejHkuerUwQlVs6pzzH+6Fixs zntSAizDgwOPoaziEZePrxptY5EXhQTEL3PuG6RE33ANPAXFTDtdCOXgV5BFK6uxWfBB PShldqhr3FE8d+h6b/flm8godmjMHHsUBFrdwnyY7eIgIJUHWLor1V02OBKBZ78gLJ6z ZDsOUm8+yv7FywsW02tLxEl/0NlHL/wzSzhUvrEh4cPHwO26p6Q0XRZ5PXNQjeYKR4JT dwdk8yxUAp+J25hrE2Ndk1jyagbwiy77IT5S3OK8DyG5MnRVt1WVXPX6LqYYU46OwH+M ecbQ== X-Gm-Message-State: ALoCoQlsZqEgqCAEGDmjywCrolfF08ALaA/raOp8SvwqNULGSDmQDac6V2cFkmUv+7dphvHK1Y1l MIME-Version: 1.0 X-Received: by 10.224.88.71 with SMTP id z7mr3282122qal.94.1406854763563; Thu, 31 Jul 2014 17:59:23 -0700 (PDT) Received: by 10.140.82.38 with HTTP; Thu, 31 Jul 2014 17:59:23 -0700 (PDT) In-Reply-To: References: Date: Thu, 31 Jul 2014 17:59:23 -0700 Message-ID: Subject: Re: Juggling or swaping out the standby NameNode in a QJM / HA configuration From: Colin Kincaid Williams To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1133da306ee6f404ff86e6ae X-Virus-Checked: Checked by ClamAV on apache.org --001a1133da306ee6f404ff86e6ae Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Another error after stopping the zkfc. Do I have to take the cluster down to format ZK? [root@rhel1 conf]# sudo service hadoop-hdfs-zkfc stop Stopping Hadoop zkfc: [ OK ] stopping zkfc [root@rhel1 conf]# sudo -u hdfs zkfc -formatZK sudo: zkfc: command not found [root@rhel1 conf]# hdfs zkfc -formatZK 2014-07-31 17:49:56,792 INFO [main] tools.DFSZKFailoverController (DFSZKFailoverController.java:(140)) - Failover controller configured for NameNode NameNode at rhel1.local/10.120.5.203:8020 2014-07-31 17:49:57,002 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:zookeeper.version=3D3.4.3-cdh4.1.3--1, built on 01/27/2013 00:1= 3 GMT 2014-07-31 17:49:57,003 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:host.name=3Drhel1.local 2014-07-31 17:49:57,003 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.version=3D1.7.0_60 2014-07-31 17:49:57,003 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.vendor=3DOracle Corporation 2014-07-31 17:49:57,003 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.home=3D/usr/java/jdk1.7.0_60/jre 2014-07-31 17:49:57,003 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.class.path=3D/etc/hadoop/conf:/usr/lib/hadoop/lib/jersey-c= ore-1.8.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jax= b-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/li= b/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoo= p/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/= commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/had= oop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/us= r/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-collections-3.= 2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/= guava-11.0.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/= hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/li= b/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jersey-server-1.= 8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/ja= ckson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/= lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hado= op/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.ja= r:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-runti= me-5.5.23.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/= lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.ja= r:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/activa= tion-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/= hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5= .jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/junit-= 4.8.2.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/common= s-math-2.1.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/li= b/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/l= ib/paranamer-2.3.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/= hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:= /usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/.//hadoop-annotat= ions-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoo= p/.//hadoop-common-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop/.//hadoop-commo= n-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/had= oop/.//hadoop-auth-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/u= sr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib= /hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mappe= r-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/l= ib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr= /lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.= cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdf= s/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:= /usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/= lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/= hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94= .jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/c= ommons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/li= b/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-= util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/u= sr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/co= mmons-codec-1.4.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3.jar:= /usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoo= p-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr= /lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/= asm-3.2.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-= yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1= .8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/a= vro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/us= r/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1= .jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/= javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/li= b/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/paranamer-= 2.3.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-y= arn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-api-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-si= te-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4= .1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanag= er.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop= -yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applica= tions-distributedshell-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-ya= rn-server-web-proxy-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-= server-tests-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-s= erver-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-pr= oxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cd= h4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.1.3= .jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.1.3.= jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-0.20-map= reduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/h= adoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6= .5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-m= apreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapred= uce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging= -1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/had= oop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/j= etty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.j= ar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20= -mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-= api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapr= educe/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-c= ompiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.j= ar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.= 20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/comm= ons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1= .6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce= /lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-d= igester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-map= reduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.= jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/us= r/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop= -0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib= /avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snap= py-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapr= educe/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-a= ll-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/h= adoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/= lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.ja= r:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hado= op-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/li= b/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairsched= uler-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-exampl= es-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr= 1-cdh4.1.3-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant-2.0.0-mr1-c= dh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoo= p-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-core.jar:/usr/lib/hadoop-0.20= -mapreduce/.//hadoop-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tool= s.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core-2.0.0-mr1-cdh4.1.3.jar:= /usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-examples.jar:/u= sr/lib/hadoop-0.20-mapreduce/.//hadoop-test-2.0.0-mr1-cdh4.1.3.jar:/usr/lib= /hadoop-0.20-mapreduce/.//hadoop-tools-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hado= op-0.20-mapreduce/.//hadoop-examples.jar:/usr/lib/hadoop-0.20-mapreduce/.//= hadoop-2.0.0-mr1-cdh4.1.3-tools.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoo= p-2.0.0-mr1-cdh4.1.3-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core= .jar 2014-07-31 17:49:57,004 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.library.path=3D//usr/lib/hadoop/lib/native 2014-07-31 17:49:57,004 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=3D/tmp 2014-07-31 17:49:57,004 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler=3D 2014-07-31 17:49:57,004 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=3DLinux 2014-07-31 17:49:57,005 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=3Damd64 2014-07-31 17:49:57,005 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=3D2.6.32-358.el6.x86_64 2014-07-31 17:49:57,005 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=3Droot 2014-07-31 17:49:57,005 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=3D/root 2014-07-31 17:49:57,005 INFO [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=3D/etc/hbase/conf.golden_apple 2014-07-31 17:49:57,015 INFO [main] zookeeper.ZooKeeper (ZooKeeper.java:(433)) - Initiating client connection, connectString=3Drhel1.local:2181,rhel6.local:2181,rhel2.local:2181 sessionTimeout=3D5000 watcher=3Dnull 2014-07-31 17:49:57,040 INFO [main-SendThread(rhel6.local:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(958)) - Opening socket connection to server rhel6.local/10.120.5.247:2181. Will not attempt to authenticate using SASL (unknown error) 2014-07-31 17:49:57,047 INFO [main-SendThread(rhel6.local:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(850)) - Socket connection established to rhel6.local/10.120.5.247:2181, initiating session 2014-07-31 17:49:57,050 INFO [main-SendThread(rhel6.local:2181)] zookeeper.ClientCnxn (ClientCnxn.java:run(1065)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2014-07-31 17:49:57,989 INFO [main] zookeeper.ZooKeeper (ZooKeeper.java:close(679)) - Session: 0x0 closed 2014-07-31 17:49:57,989 INFO [main-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(511)) - EventThread shut down Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/golden-apple' at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyEl= ector.java:263) at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.jav= a:258) at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:1= 97) at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.j= ava:58) at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:1= 65) at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:1= 61) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.j= ava:452) at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:161= ) at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverCont= roller.java:175) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =3D ConnectionLoss for /hadoop-ha/golden-apple at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049) at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyEl= ector.java:261) ... 8 more On Thu, Jul 31, 2014 at 5:56 PM, Colin Kincaid Williams wrote: > On 3) Run hdfs zkfc -formatZK in my test environment, I get a Warning > then an error > > WARNING: Before proceeding, ensure that all HDFS services and > failover controllers are stopped! > > > the complete output: > > sudo hdfs zkfc -formatZK > 2014-07-31 17:43:07,952 INFO [main] tools.DFSZKFailoverController > (DFSZKFailoverController.java:(140)) - Failover controller configur= ed > for NameNode NameNode at rhel1.local/10.120.5.203:8020 > 2014-07-31 17:43:08,128 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client > environment:zookeeper.version=3D3.4.3-cdh4.1.3--1, built on 01/27/2013 00= :13 > GMT > 2014-07-31 17:43:08,129 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:host.name=3Drhel1.loc= al > 2014-07-31 17:43:08,129 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:java.version=3D1.7.0_= 60 > 2014-07-31 17:43:08,129 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:java.vendor=3DOracle > Corporation > 2014-07-31 17:43:08,129 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client > environment:java.home=3D/usr/java/jdk1.7.0_60/jre > 2014-07-31 17:43:08,129 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client > environment:java.class.path=3D/etc/hadoop/conf:/usr/lib/hadoop/lib/jersey= -core-1.8.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/j= axb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/= lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/had= oop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/li= b/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/h= adoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/= usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-collections-= 3.2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/li= b/guava-11.0.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/li= b/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/= lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jersey-server-= 1.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/= jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoo= p/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/ha= doop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.= jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-run= time-5.5.23.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoo= p/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.= jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/acti= vation-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/li= b/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-lang-2= .5.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/juni= t-4.8.2.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/comm= ons-math-2.1.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/= lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop= /lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/li= b/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.ja= r:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/.//hadoop-annot= ations-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/had= oop/.//hadoop-common-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop/.//hadoop-com= mon-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/h= adoop/.//hadoop-auth-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:= /usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/l= ib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-map= per-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs= /lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/u= sr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.2= 6.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-h= dfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.ja= r:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop-hdf= s/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/li= b/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.= 94.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib= /commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/= lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jett= y-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:= /usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/= commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3.ja= r:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3-tests.jar:/usr/lib/had= oop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/u= sr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/li= b/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.4.Final.jar:/usr/lib/hadoo= p-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/jersey-guice= -1.8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib= /avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/= usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2= .1.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/li= b/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/= lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/paraname= r-2.3.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop= -yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn= /.//hadoop-yarn-api-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-= site-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cd= h4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn= /.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodeman= ager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hado= op-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn= /.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-appli= cations-distributedshell-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-= yarn-server-web-proxy-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yar= n-server-tests-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn= -server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-= proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-= cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.1= .3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.1.= 3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-0.20-m= apreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib= /hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapre= duce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1= .6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/= hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20= -mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.j= ar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapr= educe/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-loggi= ng-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/h= adoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib= /jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/= lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.= 20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servle= t-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/= hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-ma= preduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper= -compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-= 0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/co= mmons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools= -1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/= hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons= -digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/u= sr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/= hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-m= apreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.= 1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/= usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hado= op-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/l= ib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/sn= appy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/u= sr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-ma= preduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito= -all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib= /hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduc= e/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.= jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/= lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairsch= eduler-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-exam= ples-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-= mr1-cdh4.1.3-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant-2.0.0-mr1= -cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant.jar:/usr/lib/had= oop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-core.jar:/usr/lib/hadoop-0.= 20-mapreduce/.//hadoop-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-to= ols.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core-2.0.0-mr1-cdh4.1.3.ja= r:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-examples.jar:= /usr/lib/hadoop-0.20-mapreduce/.//hadoop-test-2.0.0-mr1-cdh4.1.3.jar:/usr/l= ib/hadoop-0.20-mapreduce/.//hadoop-tools-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/ha= doop-0.20-mapreduce/.//hadoop-examples.jar:/usr/lib/hadoop-0.20-mapreduce/.= //hadoop-2.0.0-mr1-cdh4.1.3-tools.jar:/usr/lib/hadoop-0.20-mapreduce/.//had= oop-2.0.0-mr1-cdh4.1.3-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-co= re.jar > 2014-07-31 17:43:08,130 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client > environment:java.library.path=3D//usr/lib/hadoop/lib/native > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=3D/tmp > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:java.compiler=3D > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:os.name=3DLinux > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:os.arch=3Damd64 > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client > environment:os.version=3D2.6.32-358.el6.x86_64 > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:user.name=3Droot > 2014-07-31 17:43:08,138 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client environment:user.home=3D/root > 2014-07-31 17:43:08,139 INFO [main] zookeeper.ZooKeeper > (Environment.java:logEnv(100)) - Client > environment:user.dir=3D/etc/hbase/conf.golden_apple > 2014-07-31 17:43:08,149 INFO [main] zookeeper.ZooKeeper > (ZooKeeper.java:(433)) - Initiating client connection, > connectString=3Drhel1.local:2181,rhel6.local:2181,rhel2.local:2181 > sessionTimeout=3D5000 watcher=3Dnull > 2014-07-31 17:43:08,170 INFO [main-SendThread(rhel2.local:2181)] > zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(958)) - Opening > socket connection to server rhel2.local/10.120.5.25:2181. Will not > attempt to authenticate using SASL (unknown error) > 2014-07-31 17:43:08,184 INFO [main-SendThread(rhel2.local:2181)] > zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(850)) - Socket > connection established to rhel2.local/10.120.5.25:2181, initiating sessio= n > 2014-07-31 17:43:08,262 INFO [main-SendThread(rhel2.local:2181)] > zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1187)) - Session > establishment complete on server rhel2.local/10.120.5.25:2181, sessionid > =3D 0x3478900fbb40019, negotiated timeout =3D 5000 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > The configured parent znode /hadoop-ha/golden-apple already exists. > Are you sure you want to clear all failover information from > ZooKeeper? > WARNING: Before proceeding, ensure that all HDFS services and > failover controllers are stopped! > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Proceed formatting /hadoop-ha/golden-apple? (Y or N) 2014-07-31 > 17:43:08,268 INFO [main-EventThread] ha.ActiveStandbyElector > (ActiveStandbyElector.java:processWatchEvent(538)) - Session connected. > Y > 2014-07-31 17:43:45,025 INFO [main] ha.ActiveStandbyElector > (ActiveStandbyElector.java:clearParentZNode(314)) - Recursively deleting > /hadoop-ha/golden-apple from ZK... > 2014-07-31 17:43:45,098 ERROR [main] ha.ZKFailoverController > (ZKFailoverController.java:formatZK(266)) - Unable to clear zk parent zno= de > java.io.IOException: Couldn't clear parent znode /hadoop-ha/golden-apple > at > org.apache.hadoop.ha.ActiveStandbyElector.clearParentZNode(ActiveStandbyE= lector.java:324) > at > org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.j= ava:264) > at > org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java= :197) > at > org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController= .java:58) > at > org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java= :165) > at > org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java= :161) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil= .java:452) > at > org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:1= 61) > at > org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverCo= ntroller.java:175) > Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: > KeeperErrorCode =3D Directory not empty for /hadoop-ha/golden-apple > at org.apache.zookeeper.KeeperException.create(KeeperException.java:125) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:868) > at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:54) > at > org.apache.hadoop.ha.ActiveStandbyElector$1.run(ActiveStandbyElector.java= :319) > at > org.apache.hadoop.ha.ActiveStandbyElector$1.run(ActiveStandbyElector.java= :316) > at > org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyEl= ector.java:934) > at > org.apache.hadoop.ha.ActiveStandbyElector.clearParentZNode(ActiveStandbyE= lector.java:316) > ... 8 more > 2014-07-31 17:43:45,119 INFO [main-EventThread] zookeeper.ClientCnxn > (ClientCnxn.java:run(511)) - EventThread shut down > 2014-07-31 17:43:45,119 INFO [main] zookeeper.ZooKeeper > (ZooKeeper.java:close(679)) - Session: 0x3478900fbb40019 closed > > > > On Thu, Jul 31, 2014 at 1:56 PM, Colin Kincaid Williams > wrote: > >> Thanks! I will give this a shot. >> >> >> >> On Thu, Jul 31, 2014 at 1:12 PM, Bryan Beaudreault < >> bbeaudreault@hubspot.com> wrote: >> >>> We've done this a number of times without issue. Here's the general >>> flow: >>> >>> 1) Shutdown namenode and zkfc on SNN >>> 2) Stop zkfc on ANN (ANN will remain active because there is no other >>> zkfc instance running to fail over to) >>> 3) Run hdfs zkfc -formatZK on ANN >>> 4) Start zkfc on ANN (will sync up with ANN and write state to zk) >>> 5) Push new configs to the new SNN, bootstrap namenode there >>> 6) Start namenode and zkfc on SNN >>> 7) Push updated configs to all other hdfs services (datanodes, etc) >>> 8) Restart hbasemaster if you are running hbase, jobtracker for MR >>> 9) Rolling restart datanodes >>> 10) Done >>> >>> You'll have to handle any other consumers of DFSClient, like your own >>> code or other apache projects. >>> >>> >>> >>> On Thu, Jul 31, 2014 at 3:35 PM, Colin Kincaid Williams >>> wrote: >>> >>>> Hi Jing, >>>> >>>> Thanks for the response. I will try this out, and file an Apache jira. >>>> >>>> Best, >>>> >>>> Colin Williams >>>> >>>> >>>> On Thu, Jul 31, 2014 at 11:19 AM, Jing Zhao >>>> wrote: >>>> >>>>> Hi Colin, >>>>> >>>>> I guess currently we may have to restart almost all the >>>>> daemons/services in order to swap out a standby NameNode (SBN): >>>>> >>>>> 1. The current active NameNode (ANN) needs to know the new SBN since >>>>> in the current implementation the SBN tries to send rollEditLog RPC r= equest >>>>> to ANN periodically (thus if a NN failover happens later, the origina= l ANN >>>>> needs to send this RPC to the correct NN). >>>>> 2. Looks like the DataNode currently cannot do real refreshment for >>>>> NN. Look at the code in BPOfferService: >>>>> >>>>> void refreshNNList(ArrayList addrs) throws >>>>> IOException { >>>>> Set oldAddrs =3D Sets.newHashSet(); >>>>> for (BPServiceActor actor : bpServices) { >>>>> oldAddrs.add(actor.getNNSocketAddress()); >>>>> } >>>>> Set newAddrs =3D Sets.newHashSet(addrs); >>>>> >>>>> if (!Sets.symmetricDifference(oldAddrs, newAddrs).isEmpty()) { >>>>> // Keep things simple for now -- we can implement this at a >>>>> later date. >>>>> throw new IOException( >>>>> "HA does not currently support adding a new standby to a >>>>> running DN. " + >>>>> "Please do a rolling restart of DNs to reconfigure the list >>>>> of NNs."); >>>>> } >>>>> } >>>>> >>>>> 3. If you're using automatic failover, you also need to update the >>>>> configuration of the ZKFC on the current ANN machine, since ZKFC will= do >>>>> gracefully fencing by sending RPC to the other NN. >>>>> 4. Looks like we do not need to restart JournalNodes for the new SBN >>>>> but I have not tried before. >>>>> >>>>> Thus in general we may still have to restart all the services >>>>> (except JNs) and update their configurations. But this may be a rolli= ng >>>>> restart process I guess: >>>>> 1. Shutdown the old SBN, bootstrap the new SBN, and start the new SBN= . >>>>> 2. Keep the ANN and its corresponding ZKFC running, do a rolling >>>>> restart of all the DN to update their configurations >>>>> 3. After restarting all the DN, stop ANN and the ZKFC, and update >>>>> their configuration. The new SBN should become active. >>>>> >>>>> I have not tried the upper steps, thus please let me know if this >>>>> works or not. And I think we should also document the correct steps i= n >>>>> Apache. Could you please file an Apache jira? >>>>> >>>>> Thanks, >>>>> -Jing >>>>> >>>>> >>>>> >>>>> On Thu, Jul 31, 2014 at 9:37 AM, Colin Kincaid Williams < >>>>> discord@uw.edu> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I'm trying to swap out a standby NameNode in a QJM / HA >>>>>> configuration. I believe the steps to achieve this would be somethin= g >>>>>> similar to: >>>>>> >>>>>> Use the Bootstrap standby command to prep the replacment standby. Or >>>>>> rsync if the command fails. >>>>>> >>>>>> Somehow update the datanodes, so they push the heartbeat / journal t= o >>>>>> the new standby >>>>>> >>>>>> Update the xml configuration on all nodes to reflect the replacment >>>>>> standby. >>>>>> >>>>>> Start the replacment standby >>>>>> >>>>>> Use some hadoop command to refresh the datanodes to the new NameNode >>>>>> configuration. >>>>>> >>>>>> I am not sure how to deal with the Journal switch, or if I am going >>>>>> about this the right way. Can anybody give me some suggestions here? >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Colin Williams >>>>>> >>>>>> >>>>> >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to which it is addressed and may contain information that is >>>>> confidential, privileged and exempt from disclosure under applicable = law. >>>>> If the reader of this message is not the intended recipient, you are = hereby >>>>> notified that any printing, copying, dissemination, distribution, >>>>> disclosure or forwarding of this communication is strictly prohibited= . If >>>>> you have received this communication in error, please contact the sen= der >>>>> immediately and delete it from your system. Thank You. >>>> >>>> >>>> >>> >> > --001a1133da306ee6f404ff86e6ae Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Another error after stopping the zkfc. Do I have to take t= he cluster down to format ZK?

[root@rhel1 conf]# su= do service hadoop-hdfs-zkfc stop
Stopping Hadoop zkfc: =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[ =C2=A0OK =C2=A0]
stopping zkfc
[root@rhel1 conf]# sudo -u hdfs zkfc -formatZK=
sudo: zkfc: command not found
[root@rhel1 conf]# hdfs = zkfc -formatZK
2014-07-31 17:49:56,792 INFO =C2=A0[main] tools.DF= SZKFailoverController (DFSZKFailoverController.java:<init>(140)) - Fa= ilover controller configured for NameNode NameNode at rhel1.local/10.120.5.203:8020
2014-07-31 17:49:57,002 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:zookeeper.version=3D3.4.3-cdh4.= 1.3--1, built on 01/27/2013 00:13 GMT
2014-07-31 17:49:57,003 INF= O =C2=A0[main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client = environment:host.name=3Drhel1.local
2014-07-31 17:49:57,003 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.version=3D1.7.0_60
2014-07-31 17:49:57,003 INFO =C2=A0[main] zookeeper.ZooKeeper (Environme= nt.java:logEnv(100)) - Client environment:java.vendor=3DOracle Corporation<= /div>
2014-07-31 17:49:57,003 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.home=3D/usr/java/jdk1.7.0_= 60/jre
2014-07-31 17:49:57,003 INFO =C2=A0[main] zookeeper.ZooKee= per (Environment.java:logEnv(100)) - Client environment:java.class.path=3D/= etc/hadoop/conf:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib= /commons-cli-1.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/had= oop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/u= sr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hado= op/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/= lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/li= b/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jsp-api-2.1.ja= r:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/jac= kson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoo= p/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:= /usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/avro-1.7.1.clouder= a.2.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jaspe= r-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/= hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib= /hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.= 6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jackson-x= c-1.8.8.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/l= ib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/li= b/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-h= ttpclient-3.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/li= b/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.= 7.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/snappy= -java-1.0.4.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/s= tax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/= lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/had= oop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/h= adoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:= /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/commons-cod= ec-1.4.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.1.3.jar:/usr/li= b/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.1= .3-tests.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.1.3.jar:/usr/lib/h= adoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.1= .3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/= hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.= 2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop= -hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/ha= doop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.5= 2.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-= hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.ja= r:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/zookee= per-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/li= b/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1= .8.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/c= ommons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hado= op-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtim= e-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr= /lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf= -java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/ha= doop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/.//hadoop-= hdfs-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/= lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/jackson-ma= pper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yar= n/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8= .jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib= /log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/avro-1.7.1.cloudera.2.jar:/usr/l= ib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0= .jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/a= opalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/had= oop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-yarn/lib/guice-servlet= -3.0.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/li= b/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications= -distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.1.= 3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.1.3.jar:/usr/lib/= hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib= /hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//= hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tes= ts-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar= :/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cd= h4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.= 1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.1.3-tes= ts.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/= lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/./= /hadoop-yarn-server-resourcemanager-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn= /.//hadoop-yarn-server-common-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//ha= doop-yarn-server-nodemanager-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//had= oop-yarn-common.jar:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-= mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-co= ntrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-map= reduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson= -mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/u= sr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce= /lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/l= ib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20= -mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0= .52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jac= kson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.2= 0-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.= 7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/had= oop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/= lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-m= apreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson= -xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:= /usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop= -0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/= commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons= -httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/l= ib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapred= uce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib= /commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lan= g-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/= stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:= /usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.= 20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-a= pi-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduc= e/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-js= on-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.3.jar:/us= r/lib/hadoop-0.20-mapreduce/.//hadoop-examples-2.0.0-mr1-cdh4.1.3.jar:/usr/= lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-ant.jar:/usr/lib/had= oop-0.20-mapreduce/.//hadoop-ant-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.2= 0-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.= 0-mr1-cdh4.1.3-core.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test.jar:/= usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools.jar:/usr/lib/hadoop-0.20-mapr= educe/.//hadoop-core-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/= .//hadoop-2.0.0-mr1-cdh4.1.3-examples.jar:/usr/lib/hadoop-0.20-mapreduce/./= /hadoop-test-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoo= p-tools-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-exa= mples.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-tools= .jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-test.jar:/= usr/lib/hadoop-0.20-mapreduce/.//hadoop-core.jar
2014-07-31 17:49:57,004 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.library.path=3D//usr/lib/h= adoop/lib/native
2014-07-31 17:49:57,004 INFO =C2=A0[main] zookee= per.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.t= mpdir=3D/tmp
2014-07-31 17:49:57,004 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.compiler=3D<NA>
2014-07-31 17:49:57,004 INFO =C2=A0[main] zookeeper.ZooKeeper (Enviro= nment.java:logEnv(100)) - Client environment:os.= name=3DLinux
2014-07-31 17:49:57,005 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:os.arch=3Damd64
2014-= 07-31 17:49:57,005 INFO =C2=A0[main] zookeeper.ZooKeeper (Environment.java:= logEnv(100)) - Client environment:os.version=3D2.6.32-358.el6.x86_64
2014-07-31 17:49:57,005 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:us= er.name=3Droot
2014-07-31 17:49:57,005 INFO =C2=A0[main] zook= eeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.ho= me=3D/root
2014-07-31 17:49:57,005 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:user.dir=3D/etc/hbase/conf.gold= en_apple
2014-07-31 17:49:57,015 INFO =C2=A0[main] zookeeper.ZooK= eeper (ZooKeeper.java:<init>(433)) - Initiating client connection, co= nnectString=3Drhel1.local:2181,rhel6.local:2181,rhel2.local:2181 sessionTim= eout=3D5000 watcher=3Dnull
2014-07-31 17:49:57,040 INFO =C2=A0[main-SendThread(rhel6.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(958)) - Opening sock= et connection to server rhel6.local/10= .120.5.247:2181. Will not attempt to authenticate using SASL (unknown e= rror)
2014-07-31 17:49:57,047 INFO =C2=A0[main-SendThread(rhel6.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(850)) - Socket conne= ction established to rhel6.local/10.12= 0.5.247:2181, initiating session
2014-07-31 17:49:57,050 INFO =C2=A0[main-SendThread(rhel6.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:run(1065)) - Unable to read addition= al data from server sessionid 0x0, likely server has closed socket, closing= socket connection and attempting reconnect
2014-07-31 17:49:57,989 INFO =C2=A0[main] zookeeper.ZooKeeper (ZooKeep= er.java:close(679)) - Session: 0x0 closed
2014-07-31 17:49:57,989= INFO =C2=A0[main-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(51= 1)) - EventThread shut down
Exception in thread "main" java.io.IOException: Couldn't= determine existence of znode '/hadoop-ha/golden-apple'
<= span class=3D"" style=3D"white-space:pre"> at org.apache.hadoop.ha.A= ctiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
at org.apache.hadoo= p.ha.ZKFailoverController.formatZK(ZKFailoverController.java:258)
at org.apache.hadoop.ha= .ZKFailoverController.doRun(ZKFailoverController.java:197)
at org.apache.hadoo= p.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
at org.apache.hadoop.h= a.ZKFailoverController$1.run(ZKFailoverController.java:165)
at org.apache.hadoo= p.ha.ZKFailoverController$1.run(ZKFailoverController.java:161)
at org.apache.hadoop.secur= ity.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:452)
at org.apache.hadoo= p.ha.ZKFailoverController.run(ZKFailoverController.java:161)
at org.apache.hadoop.hdfs.to= ols.DFSZKFailoverController.main(DFSZKFailoverController.java:175)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossExceptio= n: KeeperErrorCode =3D ConnectionLoss for /hadoop-ha/golden-apple
at org.apache.zookeeper= .KeeperException.create(KeeperException.java:99)
at org.apache.zooke= eper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.e= xists(ZooKeeper.java:1021)
at org.apache.zooke= eper.ZooKeeper.exists(ZooKeeper.java:1049)
at org.apache.hadoop.ha.ActiveStandbyElector.p= arentZNodeExists(ActiveStandbyElector.java:261)
... 8 more



On Thu, Jul 31, 2014 at 5:56 PM, Colin Kincaid Williams <discord@u= w.edu> wrote:
On =C2=A03) Run hdfs z= kfc -formatZK in my test environment, I get a Warning then an error

WARNING: Before proceeding, ensure that all HDFS services a= nd
failover controllers are stopped!


the complete output:
sudo hdfs zkfc -formatZK
2014-07-31 17:43:07,952 INFO= =C2=A0[main] tools.DFSZKFailoverController (DFSZKFailoverController.java:&= lt;init>(140)) - Failover controller configured for NameNode NameNode at= rhel1.local/10.120.= 5.203:8020
2014-07-31 17:43:08,128 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:zookeeper.version=3D3.4.3-cdh4.= 1.3--1, built on 01/27/2013 00:13 GMT
2014-07-31 17:43:08,129 INF= O =C2=A0[main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client = environment:host.name=3D= rhel1.local
2014-07-31 17:43:08,129 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.version=3D1.7.0_60
2014-07-31 17:43:08,129 INFO =C2=A0[main] zookeeper.ZooKeeper (Environme= nt.java:logEnv(100)) - Client environment:java.vendor=3DOracle Corporation<= /div>
2014-07-31 17:43:08,129 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.home=3D/usr/java/jdk1.7.0_= 60/jre
2014-07-31 17:43:08,129 INFO =C2=A0[main] zookeeper.ZooKee= per (Environment.java:logEnv(100)) - Client environment:java.class.path=3D/= etc/hadoop/conf:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib= /commons-cli-1.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/had= oop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/u= sr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hado= op/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/= lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/li= b/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jsp-api-2.1.ja= r:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/jac= kson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoo= p/lib/zookeeper-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:= /usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/avro-1.7.1.clouder= a.2.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jaspe= r-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/= hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib= /hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.= 6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jackson-x= c-1.8.8.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/l= ib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/li= b/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-h= ttpclient-3.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/li= b/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.= 7.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/snappy= -java-1.0.4.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/s= tax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/= lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/had= oop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/h= adoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:= /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/commons-cod= ec-1.4.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.1.3.jar:/usr/li= b/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.1= .3-tests.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.1.3.jar:/usr/lib/h= adoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.1= .3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/= hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.= 2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop= -hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/ha= doop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.5= 2.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-= hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.ja= r:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/zookee= per-3.4.3-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/li= b/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1= .8.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/c= ommons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hado= op-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtim= e-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr= /lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf= -java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/ha= doop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-hdfs/.//hadoop-= hdfs-2.0.0-cdh4.1.3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/= lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/jackson-ma= pper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yar= n/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8= .jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib= /log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/avro-1.7.1.cloudera.2.jar:/usr/l= ib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0= .jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/a= opalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/had= oop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-yarn/lib/guice-servlet= -3.0.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/li= b/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications= -distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.1.= 3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.1.3.jar:/usr/lib/= hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.= //hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib= /hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//= hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tes= ts-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar= :/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cd= h4.1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.= 1.3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.1.3-tes= ts.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/= lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/./= /hadoop-yarn-server-resourcemanager-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn= /.//hadoop-yarn-server-common-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//ha= doop-yarn-server-nodemanager-2.0.0-cdh4.1.3.jar:/usr/lib/hadoop-yarn/.//had= oop-yarn-common.jar:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-= mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-co= ntrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-map= reduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson= -mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/u= sr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce= /lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/l= ib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20= -mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0= .52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr= /lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jac= kson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.2= 0-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.= 7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar= :/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/had= oop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapredu= ce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/= lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-m= apreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson= -xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:= /usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop= -0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/= commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons= -httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/l= ib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapred= uce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib= /commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lan= g-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2= .jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/ha= doop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/= stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:= /usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.= 20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-a= pi-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduc= e/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-js= on-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/li= b/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.3.jar:/us= r/lib/hadoop-0.20-mapreduce/.//hadoop-examples-2.0.0-mr1-cdh4.1.3.jar:/usr/= lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-ant.jar:/usr/lib/had= oop-0.20-mapreduce/.//hadoop-ant-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.2= 0-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.= 0-mr1-cdh4.1.3-core.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test.jar:/= usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools.jar:/usr/lib/hadoop-0.20-mapr= educe/.//hadoop-core-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/= .//hadoop-2.0.0-mr1-cdh4.1.3-examples.jar:/usr/lib/hadoop-0.20-mapreduce/./= /hadoop-test-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoo= p-tools-2.0.0-mr1-cdh4.1.3.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-exa= mples.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-tools= .jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-2.0.0-mr1-cdh4.1.3-test.jar:/= usr/lib/hadoop-0.20-mapreduce/.//hadoop-core.jar
2014-07-31 17:43:08,130 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.library.path=3D//usr/lib/h= adoop/lib/native
2014-07-31 17:43:08,138 INFO =C2=A0[main] zookee= per.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.t= mpdir=3D/tmp
2014-07-31 17:43:08,138 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:java.compiler=3D<NA>
2014-07-31 17:43:08,138 INFO =C2=A0[main] zookeeper.ZooKeeper (Enviro= nment.java:logEnv(100)) - Client environment:os.name=3DLinux
2014-07-31 17:43:08,138 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:os.arch=3Damd64
2014-= 07-31 17:43:08,138 INFO =C2=A0[main] zookeeper.ZooKeeper (Environment.java:= logEnv(100)) - Client environment:os.version=3D2.6.32-358.el6.x86_64
2014-07-31 17:43:08,138 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:user.name=3Droot
2014-07-31 17:43:08,138 INFO= =C2=A0[main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client e= nvironment:user.home=3D/root
2014-07-31 17:43:08,139 INFO =C2=A0[main] zookeeper.ZooKeeper (Environ= ment.java:logEnv(100)) - Client environment:user.dir=3D/etc/hbase/conf.gold= en_apple
2014-07-31 17:43:08,149 INFO =C2=A0[main] zookeeper.ZooK= eeper (ZooKeeper.java:<init>(433)) - Initiating client connection, co= nnectString=3Drhel1.local:2181,rhel6.local:2181,rhel2.local:2181 sessionTim= eout=3D5000 watcher=3Dnull
2014-07-31 17:43:08,170 INFO =C2=A0[main-SendThread(rhel2.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(958)) - Opening sock= et connection to server rhel2.local/10.120.5.25:2181. Will not attempt to authenticate using= SASL (unknown error)
2014-07-31 17:43:08,184 INFO =C2=A0[main-SendThread(rhel2.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(850)) - Socket conne= ction established to rhel2.local/10.120.5.25:2181, initiating session
2014-07-31 17:43:08,262 INFO =C2=A0[main-SendThread(rhel2.local:2181)]= zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1187)) - Session establi= shment complete on server rhel2.local/10.120.5.25:2181, sessionid =3D 0x3478900fbb40019, neg= otiated timeout =3D 5000
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
The configured parent znode /hadoop-ha/golden-apple already exists.=
Are you sure you want to clear all failover information from
ZooKeeper?
WARNING: Before proceeding, ensure that all HDFS services and
failover controllers are stopped!
=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Proceed formatting /hadoo= p-ha/golden-apple? (Y or N) 2014-07-31 17:43:08,268 INFO =C2=A0[main-EventT= hread] ha.ActiveStandbyElector (ActiveStandbyElector.java:processWatchEvent= (538)) - Session connected.
Y
2014-07-31 17:43:45,025 INFO =C2=A0[main] ha.ActiveStandby= Elector (ActiveStandbyElector.java:clearParentZNode(314)) - Recursively del= eting /hadoop-ha/golden-apple from ZK...
2014-07-31 17:43:45,098 = ERROR [main] ha.ZKFailoverController (ZKFailoverController.java:formatZK(26= 6)) - Unable to clear zk parent znode
java.io.IOException: Couldn't clear parent znode /hadoop-ha/golden= -apple
at org.apache= .hadoop.ha.ActiveStandbyElector.clearParentZNode(ActiveStandbyElector.java:= 324)
at org.apache.hadoop.ha.Z= KFailoverController.formatZK(ZKFailoverController.java:264)
at org.apache.hadoop.ha.ZKFailoverC= ontroller.doRun(ZKFailoverController.java:197)
at org.apache.hadoop.ha.Z= KFailoverController.access$000(ZKFailoverController.java:58)
at org.apache.hadoop.ha.ZKFailover= Controller$1.run(ZKFailoverController.java:165)
at org.apache.hadoop.ha.Z= KFailoverController$1.run(ZKFailoverController.java:161)
at org.apache.hadoop.security.Security= Util.doAsLoginUserOrFatal(SecurityUtil.java:452)
at org.apache.hadoop.ha.Z= KFailoverController.run(ZKFailoverController.java:161)
at org.apache.hadoop.hdfs.tools.DFSZKFai= loverController.main(DFSZKFailoverController.java:175)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: Kee= perErrorCode =3D Directory not empty for /hadoop-ha/golden-apple
= at org.apache.zookeeper.Keeper= Exception.create(KeeperException.java:125)
at org.apache.zookeeper.K= eeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeep= er.java:868)
at org.apache.zookeeper.Z= KUtil.deleteRecursive(ZKUtil.java:54)
at org.apache.hadoop.ha.ActiveStandbyElector$1.run(Active= StandbyElector.java:319)
at org.apache.hadoop.ha.A= ctiveStandbyElector$1.run(ActiveStandbyElector.java:316)
at org.apache.hadoop.ha.ActiveStandbyE= lector.zkDoWithRetries(ActiveStandbyElector.java:934)
at org.apache.hadoop.ha.A= ctiveStandbyElector.clearParentZNode(ActiveStandbyElector.java:316)
... 8 more
2014-07-31 17:43:45,119 INFO =C2=A0[main-EventThread] zookeeper.ClientCnxn = (ClientCnxn.java:run(511)) - EventThread shut down
2014-07-31 17:= 43:45,119 INFO =C2=A0[main] zookeeper.ZooKeeper (ZooKeeper.java:close(679))= - Session: 0x3478900fbb40019 closed



On Thu, Jul 31, 2014 at 1:5= 6 PM, Colin Kincaid Williams <discord@uw.edu> wrote:
Thanks! I will give this a = shot.



On Thu, Jul 31, 2014 at 1:12 PM, Bryan B= eaudreault <bbeaudreault@hubspot.com> wrote:
We've done this a numbe= r of times without issue. =C2=A0Here's the general flow:

=
1) Shutdown namenode and zkfc on SNN
2) Stop zkfc on ANN =C2=A0(ANN will remain active because there is no = other zkfc instance running to fail over to)
3) Run hdfs zkfc -formatZK on ANN
4) Start zkfc on ANN (will= sync up with ANN and write state to zk)
5) Push new configs to t= he new SNN, bootstrap namenode there
6) Start namenode and zkfc o= n SNN
7) Push updated configs to all other hdfs services (datanodes, etc)
8) Restart hbasemaster if you are running hbase, jobtracker for MR<= /div>
9) Rolling restart datanodes
10) Done

You'll have to handle any other consumers of DFSClient, like= your own code or other apache projects.



On Thu, Jul 31, 2014 at 3:35 PM, Colin Kincaid Williams &= lt;discord@uw.edu&g= t; wrote:
Hi Jing,=C2=A0

Thanks for the response. I will try this out, and file an Apache jir= a.

Best,

Colin Williams


On Thu, Jul 31, 2014 at 11:19 AM, Jing Z= hao <jing@hortonworks.com> wrote:
Hi Colin,

=C2=A0 =C2=A0 I guess current= ly we may have to restart almost all the daemons/services in order to swap = out a standby NameNode (SBN):

1. The current activ= e NameNode (ANN) needs to know the new SBN since in the current implementat= ion the SBN tries to send rollEditLog RPC request to ANN periodically (thus= if a NN failover happens later, the original ANN needs to send this RPC to= the correct NN).
2. Looks like the DataNode currently cannot do real refreshment for NN= . Look at the code in BPOfferService:

=C2=A0 = void refreshNNList(ArrayList<InetSocketAddress> addrs) throws IOExcep= tion {
=C2=A0 =C2=A0 Set<InetSocketAddress> oldAddrs =3D Sets.newHashSe= t();
=C2=A0 =C2=A0 for (BPServiceActor actor : bpServices) {
=C2=A0 =C2=A0 =C2=A0 oldAddrs.add(actor.getNNSocketAddress());
<= div>=C2=A0 =C2=A0 }
=C2=A0 =C2=A0 Set<InetSocketAddress> ne= wAddrs =3D Sets.newHashSet(addrs);
=C2=A0 =C2=A0=C2=A0
=C2=A0 =C2=A0 if (!Sets.symmetricDiffere= nce(oldAddrs, newAddrs).isEmpty()) {
=C2=A0 =C2=A0 =C2=A0 // Keep= things simple for now -- we can implement this at a later date.
= =C2=A0 =C2=A0 =C2=A0 throw new IOException(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "HA does not currently support addi= ng a new standby to a running DN. " +
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 "Please do a rolling restart of DNs to reconfigure the l= ist of NNs.");
=C2=A0 =C2=A0 }
=C2=A0 }

3. If you're using automatic failover, you al= so need to update the configuration of the ZKFC on the current ANN machine,= since ZKFC will do gracefully fencing by sending RPC to the other NN.
4. Looks like we do not need to restart JournalNodes for the new SBN b= ut I have not tried before.

=C2=A0 =C2=A0 Thus in = general we may still have to restart all the services (except JNs) and upda= te their configurations. But this may be a rolling restart process I guess:=
1. Shutdown the old SBN, bootstrap the new SBN, and start the new SBN.=
2. Keep the ANN and its corresponding ZKFC running, do a rolling= restart of all the DN to update their configurations
3. After re= starting all the DN, stop ANN and the ZKFC, and update their configuration.= The new SBN should become active.

=C2=A0 =C2=A0 I have not tried the upper steps, thus pl= ease let me know if this works or not. And I think we should also document = the correct steps in Apache. Could you please file an Apache jira?

Thanks,
-Jing



On Thu, Jul 31, 2= 014 at 9:37 AM, Colin Kincaid Williams <discord@uw.edu> wrote:<= br>
Hello,

I= 'm trying to swap out a standby NameNode in a QJM / HA configuration. I= believe the steps to achieve this would be something similar to:

Use the Bootstrap standby command to prep the replacmen= t standby. Or rsync if the command fails.

Somehow update the datanodes, so they push the heartbea= t / journal to the new standby

Update the xml conf= iguration on all nodes to reflect the replacment standby.

Start the replacment standby=C2=A0

Use = some hadoop command to refresh the datanodes to the new NameNode configurat= ion.

I am not sure how to deal with the Journal sw= itch, or if I am going about this the right way. Can anybody give me some s= uggestions here?=C2=A0


Regards,

= Colin Williams



CONFIDENTIALITY NOTICE
NOTICE: This= message is intended for the use of the individual or entity to which it is= addressed and may contain information that is confidential, privileged and= exempt from disclosure under applicable law. If the reader of this message= is not the intended recipient, you are hereby notified that any printing, = copying, dissemination, distribution, disclosure or forwarding of this comm= unication is strictly prohibited. If you have received this communication i= n error, please contact the sender immediately and delete it from your syst= em. Thank You.





--001a1133da306ee6f404ff86e6ae--