Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9764A9581 for ; Tue, 3 Apr 2012 08:04:19 +0000 (UTC) Received: (qmail 11996 invoked by uid 500); 3 Apr 2012 08:04:17 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 11943 invoked by uid 500); 3 Apr 2012 08:04:17 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 11912 invoked by uid 99); 3 Apr 2012 08:04:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Apr 2012 08:04:15 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sdiepend@gmail.com designates 209.85.210.169 as permitted sender) Received: from [209.85.210.169] (HELO mail-iy0-f169.google.com) (209.85.210.169) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Apr 2012 08:04:11 +0000 Received: by iajr24 with SMTP id r24so7071270iaj.14 for ; Tue, 03 Apr 2012 01:03:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=rm+jOetjGWM9aFEecK1Dg93EEtAdPAvW3ReRfdljnZM=; b=HwtjzCSPHHGEFxsq4QbSBfuygV34k2DSRyR8j/L5AVVhrPLhTFKU0+rC9fl/RFWks5 r2vOUXOHzXWiw6CPEImVRbu+WvC/oLnSkKwZyFfC+yzJhkMHpmveZTXsAzbCrDq+JzMm +nEkuc+tAwUK11i8Kv3rs+6+UKwMgtn4Kuv3u2NvzrWvg957SPnI1XPpS/S9xoitWTCq FHLj1cZ1S4BDfTQ8d9nONYf9bZzyPFko57Dts6b5OFPHPPYimDT97qGdUSM6Fa4gHN/r 0+MkwSXAbnR/RfoP9e5la8yxXNJVNA4ZR+w8oy4UUHDLvBKSEdNQtChHt3A2aOJxZ9XH viOA== MIME-Version: 1.0 Received: by 10.50.159.202 with SMTP id xe10mr7837148igb.66.1333440231176; Tue, 03 Apr 2012 01:03:51 -0700 (PDT) Received: by 10.231.243.193 with HTTP; Tue, 3 Apr 2012 01:03:51 -0700 (PDT) In-Reply-To: References: Date: Tue, 3 Apr 2012 10:03:51 +0200 Message-ID: Subject: Re: HBase starting problem From: Stijn Diependaele To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=14dae93406014ef95304bcc1bf3e X-Virus-Checked: Checked by ClamAV on apache.org --14dae93406014ef95304bcc1bf3e Content-Type: text/plain; charset=ISO-8859-1 Sorry to dig up my thread again but couldn't my problem be related to this: https://issues.apache.org/jira/browse/HADOOP-7988 ? since I have this error: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)] too. 2012/3/23 Sonal Goyal > Your logs still point to a permissions issue on /hbase, so I would start at > resolving that first thing.. > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602) > *Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: > KeeperErrorCode = AuthFailed for /hbase > at org.apache.zookeeper.**KeeperException.creat*e( > KeeperException.java:123) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > > Best Regards, > Sonal > Crux: Reporting for HBase > Nube Technologies > > > > > > > > On Thu, Mar 22, 2012 at 6:56 PM, Stijn Diependaele >wrote: > > > Thanks for your reply, > > > > Currently I have no special permissions set for dfs, I think it's all > > default. > > It's the root that's running the dfs, same for zookeeper it's also the > > root. > > But for some reason the log from the hbase master is stored in Stijn, > which > > is a normal user, I'm not sure if this has anything to do with the > problem. > > Normally it should be root that runs it so it should be logged into the > > hbase-root-master... file. > > > > logs/hbase-Stijn-master-HADOOP.domain.be.log > > > > Now I've removed the hbase.keytab and krb5.keytab and I made new ones. > Now > > I'm getting a different error: > > And it seems that some problem could be the resolving of the hostname, > but > > normally that's configured well. > > > > ..... > > ..... > > 2012-03-22 14:19:28,166 INFO org.apache.zookeeper.ClientCnxn: Session > > establishment complete on server HDP06.domain.be/10.11.12.206:2222, > > sessionid = 0x1363a8fc39d0001, negotiated timeout = 180000 > > 2012-03-22 14:19:28,194 ERROR > > org.apache.zookeeper.client.ZooKeeperSaslClient: An error: > > (java.security.PrivilegedActionException: > > javax.security.sasl.SaslException: GSS initiate failed [Caused by > > GSSException: No valid credentials provided (Mechanism level: Server not > > found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when > evaluating > > Zookeeper Quorum Member's received SASL token. This may be caused by > > Java's being unable to resolve the Zookeeper Quorum Member's hostname > > correctly. You may want to try to adding > > '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS > > environment. Zookeeper Client will go to AUTH_FAILED state. > > 2012-03-22 14:19:28,194 ERROR org.apache.zookeeper.ClientCnxn: SASL > > authentication with Zookeeper Quorum member failed: > > javax.security.sasl.SaslException: An error: > > (java.security.PrivilegedActionException: > > javax.security.sasl.SaslException: GSS initiate failed [Caused by > > GSSException: No valid credentials provided (Mechanism level: Server not > > found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when > evaluating > > Zookeeper Quorum Member's received SASL token. This may be caused by > > Java's being unable to resolve the Zookeeper Quorum Member's hostname > > correctly. You may want to try to adding > > '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS > > environment. Zookeeper Client will go to AUTH_FAILED state. > > 2012-03-22 14:19:28,229 ERROR > > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master > > java.lang.RuntimeException: Failed construction of Master: class > > org.apache.hadoop.hbase.master.HMaster > > at > > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588) > > at > > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144) > > at > > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at > > > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602) > > Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: > > KeeperErrorCode = AuthFailed for /hbase > > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:123) > > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003) > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1031) > > at > > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:189) > > at > > > > > org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:892) > > at > > > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:159) > > at > > > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:152) > > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:264) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > > at > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:525) > > at > > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1583) > > ... 5 more > > > > > > 2012/3/21 Sonal Goyal > > > > > Hi Stijn, > > > > > > You seem to be hitting permission issues with /hbase > > > > > > Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: > > > KeeperErrorCode = InvalidACL for /hbase > > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:121) > > > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > > > > > > What are your dfs user permission settings like? Which user are you > > running > > > hbase as? > > > > > > Best Regards, > > > Sonal > > > Crux: Reporting for HBase > > > Nube Technologies > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 21, 2012 at 3:49 PM, Stijn Diependaele > > >wrote: > > > > > > > I'm trying to setup HBase on top of my HDFS filesystem. > > > > I currently have 6 servers but only 3 are configured to run zookeeper > > and > > > > HBase, for testing purposes and the idea of adding the other servers > > when > > > > this works. > > > > I'm following this guide > > > http://hbase.apache.org/book/configuration.htmlto > > > > set it up. > > > > > > > > I can paste the entire log in pastebin if you want. > > > > HBase-site.xml : http://pastebin.com/Gb8UFSHj > > > > > > > > Errors in the log files: > > > > > > > > Zookeeper: > > > > ---------- > > > > > > > > logs/hbase-root-zookeeper-HADOOP.domain.be.log (master) > > > > .... > > > > .... > > > > 2012-03-21 09:58:39,576 INFO > > org.apache.zookeeper.server.ZooKeeperServer: > > > > Established session 0x36347aae9a0000 with negotiated timeout 180000 > for > > > > client /10.11.12.206:43325 > > > > 2012-03-21 09:58:39,611 WARN > org.apache.zookeeper.server.NIOServerCnxn: > > > > caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > > sessionid > > > > 0x36347aae9a0000, likely client has closed socket > > > > at > > > > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) > > > > at > > > > > > > > > > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224) > > > > at java.lang.Thread.run(Thread.java:722) > > > > 2012-03-21 09:58:39,617 INFO > org.apache.zookeeper.server.NIOServerCnxn: > > > > Closed socket connection for client /10.11.12.206:43325 which had > > > > sessionid > > > > 0x36347aae9a0000 > > > > .... > > > > .... > > > > > > > > on a znode: > > > > logs/hbase-root-zookeeper-HDP08.domain.be.log > > > > ..... > > > > ..... > > > > 2012-03-21 09:58:31,290 INFO > > > > org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to > > > majority > > > > quorums > > > > 2012-03-21 09:58:31,433 DEBUG > org.apache.hadoop.hbase.util.VersionInfo: > > > > preRegister called. > > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908 > > > , > > > > name=log4j:logger=org.apache.hadoop.hbase.util.VersionInfo > > > > 2012-03-21 09:58:31,434 DEBUG > > org.apache.hadoop.hbase.zookeeper.ZKConfig: > > > > preRegister called. > > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908 > > > , > > > > name=log4j:logger=org.apache.hadoop.hbase.zookeeper.ZKConfig > > > > 2012-03-21 09:58:31,434 DEBUG > > org.apache.hadoop.hbase.HBaseConfiguration: > > > > preRegister called. > > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908 > > > , > > > > name=log4j:logger=org.apache.hadoop.hbase.HBaseConfiguration > > > > 2012-03-21 09:58:31,435 DEBUG org.apache.hadoop.hbase: preRegister > > > called. > > > > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908, > > > > name=log4j:logger=org.apache.hadoop.hbase > > > > 2012-03-21 09:58:31,435 INFO > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum > peer > > > > 2012-03-21 09:58:31,836 INFO org.apache.zookeeper.Login: successfully > > > > logged in. > > > > 2012-03-21 09:58:31,842 INFO org.apache.zookeeper.Login: TGT refresh > > > thread > > > > started. > > > > ..... > > > > ..... > > > > a bit further down the log file: > > > > .... > > > > .... > > > > 2012-03-21 09:58:32,425 INFO > > > > org.apache.zookeeper.server.quorum.LearnerHandler: Sending snapshot > > last > > > > zxid of peer is 0x1800000012 zxid of leader is 0x1900000000sent zxid > > of > > > db > > > > as 0x1800000012 > > > > 2012-03-21 09:58:32,466 WARN > org.apache.zookeeper.server.quorum.Leader: > > > > Commiting zxid 0x1900000000 from /10.11.12.208:2888 not first! > > > > 2012-03-21 09:58:32,466 WARN > org.apache.zookeeper.server.quorum.Leader: > > > > First is 0 > > > > 2012-03-21 09:58:32,466 INFO > org.apache.zookeeper.server.quorum.Leader: > > > > Have quorum of supporters; starting up and setting last processed > zxid: > > > > 107374182400 > > > > 2012-03-21 09:58:36,505 INFO > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager: Received > > connection > > > > request /10.11.12.200:46015 > > > > .... > > > > .... > > > > > > > > HBase: > > > > ----- > > > > master: > > > > .... > > > > .... > > > > 2012-03-21 09:58:39,162 WARN org.apache.zookeeper.ClientCnxn: > Zookeeper > > > > client cannot authenticate using the Client section of the supplied > > > > configuration file: '/home/Stijn/hbase-0.92.0/conf/jaas.conf'. Will > > > > continue connection to Zookeeper server without SASL authentication, > if > > > > Zookeeper server allows it. > > > > .... > > > > .... > > > > 2012-03-21 09:58:39,313 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server HDP06.domain.be/10.11.12.206:2222, > > > > sessionid = 0x136347a9db40000, negotiated timeout = 180000 > > > > 2012-03-21 09:58:39,454 ERROR > > > > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start > > master > > > > java.lang.RuntimeException: Failed construction of Master: class > > > > org.apache.hadoop.hbase.master.HMaster > > > > at > > > > > > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103) > > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) > > > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602) > > > > Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: > > > > KeeperErrorCode = InvalidACL for /hbase > > > > at > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:121) > > > > at > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > > > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > > > > ..... > > > > ..... > > > > > > > > > > --14dae93406014ef95304bcc1bf3e--