Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A55CD10D84 for ; Wed, 5 Mar 2014 08:39:50 +0000 (UTC) Received: (qmail 12952 invoked by uid 500); 5 Mar 2014 08:39:47 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 12128 invoked by uid 500); 5 Mar 2014 08:39:46 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 12083 invoked by uid 99); 5 Mar 2014 08:39:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Mar 2014 08:39:45 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cxd3000@gmail.com designates 209.85.223.172 as permitted sender) Received: from [209.85.223.172] (HELO mail-ie0-f172.google.com) (209.85.223.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Mar 2014 08:39:41 +0000 Received: by mail-ie0-f172.google.com with SMTP id as1so756962iec.3 for ; Wed, 05 Mar 2014 00:39:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Kt/bNhWY6LJLRGN1rNmhorcRkdY9dJL2BqX8ryKB4GY=; b=aQEM/Qsf7UQS+7SbaqUXod5p5eEeVzDRwbB/ij3QS3f7qGBS/TlDJQhjf2gSID/d8A F7R5jyS0RrnjjaFUdDY1YESDGixhAazE5WLFhBuIAOirPP4i50z6A8rpI91uqa3D613H xfXUVeEg9Q3aqXKq+mG76O9vMKoupULoZwINeOwb73Vi/yKlwdyRKCUUkIroUmE/3dEk RHY3fPhp+N7yHLG6WKjiHyyBsmnwTnrtf4UwFRQGV1Mpq4nTYm7U2upe+qlQWeCt70Zu RaCzJu3u/7WsolFe7aG9cv7dR1s1BEN2bU9/nw03xqR0DGvHovo7Wq+A1dYgrKVa8hKH NcDg== MIME-Version: 1.0 X-Received: by 10.51.17.104 with SMTP id gd8mr33159802igd.48.1394008760969; Wed, 05 Mar 2014 00:39:20 -0800 (PST) Received: by 10.64.9.241 with HTTP; Wed, 5 Mar 2014 00:39:20 -0800 (PST) In-Reply-To: References: Date: Wed, 5 Mar 2014 16:39:20 +0800 Message-ID: Subject: Re: Help: HMaster getting Aborted at startup! From: Richard Chen To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=001a1135e1dc02f28304f3d7f58a X-Virus-Checked: Checked by ClamAV on apache.org --001a1135e1dc02f28304f3d7f58a Content-Type: text/plain; charset=ISO-8859-1 Thanks for the suggestion. But when I looked at the JIRA, it shows - Fix Version/s:0.98.0 , 0.96.0 So why do I need to do the patch in 0.98.0 which is supposedly having the fixes inside? But still 0.98.0 has the same error during start-up...weird... BTW, aren't the stable releases passing all the tests including integration tests? why do we still see such issues in stable releases? Thanks, Richard On Wed, Mar 5, 2014 at 11:36 AM, Rabbit's Foot wrote: > You can try to refering the following link > > https://issues.apache.org/jira/browse/HBASE-9278 > > https://issues.apache.org/jira/browse/HBASE-9497 > > > 2014-03-05 10:47 GMT+08:00 Richard Chen : > > > Hmaster getting aborted after starting ./start-hbase.sh in hbase-0.96.0 > > with hadoop 2.2.0. > > > > Tried with hbase-0.94.16 and hbase-0.98 but same result. Hmaster aborts > as > > soon as it starts. Even tried with replacing jars in hbase lib manually > as > > well as using maven but the issue is unresolved. Is there any other > > solution? > > > > Below is the corresponding hbase-hadoop-master-hadoop-master.log... > > > > 2014-02-24 10:11:27,078 INFO > > [Replication.RpcServer.handler=2,port=60000] ipc.RpcServer: > > Replication.RpcServer.handler=2,port=60000: starting > > 2014-02-24 10:11:27,565 INFO [RpcServer.handler=23,port=60000] > > ipc.RpcServer: RpcServer.handler=23,port=60000: starting > > 2014-02-24 10:11:27,970 INFO [master:hadoop-master:60000] > > mortbay.log: Logging to > > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > > org.mortbay.log.Slf4jLog > > 2014-02-24 10:11:28,172 INFO [master:hadoop-master:60000] > > http.HttpServer: Added global filter 'safety' > > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > > 2014-02-24 10:11:28,177 INFO [master:hadoop-master:60000] > > http.HttpServer: Added filter static_user_filter > > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) > > to context master > > 2014-02-24 10:11:28,177 INFO [master:hadoop-master:60000] > > http.HttpServer: Added filter static_user_filter > > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) > > to context static > > 2014-02-24 10:11:28,191 INFO [master:hadoop-master:60000] > > http.HttpServer: Jetty bound to port 60010 > > 2014-02-24 10:11:28,191 INFO [master:hadoop-master:60000] > > mortbay.log: jetty-6.1.26 > > 2014-02-24 10:11:29,227 INFO [master:hadoop-master:60000] > > mortbay.log: Started SelectChannelConnector@0.0.0.0:60010 > > 2014-02-24 10:11:29,623 INFO [master:hadoop-master:60000] > > master.ActiveMasterManager: Registered Active > > Master=hadoop-master.payoda.com,60000,1393236677609 > > 2014-02-24 10:11:29,629 INFO [master:hadoop-master:60000] > > Configuration.deprecation: fs.default.name is deprecated. Instead, use > > fs.defaultFS > > 2014-02-24 10:11:29,851 DEBUG [main-EventThread] > > master.ActiveMasterManager: A master is now available > > 2014-02-24 10:11:30,537 INFO [master:hadoop-master:60000] > > Configuration.deprecation: hadoop.native.lib is deprecated. Instead, > > use io.native.lib.available > > 2014-02-24 10:11:30,800 DEBUG [master:hadoop-master:60000] > > util.FSTableDescriptors: Current tableInfoPath = > > > > > hdfs://hadoop-master:9000/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 > > 2014-02-24 10:11:30,821 DEBUG [master:hadoop-master:60000] > > util.FSTableDescriptors: TableInfo already exists.. Skipping creation > > 2014-02-24 10:11:30,944 INFO [master:hadoop-master:60000] > > fs.HFileSystem: Added intercepting call to namenode#getBlockLocations > > so can do block reordering using class class > > org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks > > 2014-02-24 10:11:30,950 INFO [master:hadoop-master:60000] > > master.SplitLogManager: Timeout=120000, unassigned timeout=180000, > > distributedLogReplay=false > > 2014-02-24 10:11:30,956 INFO [master:hadoop-master:60000] > > master.SplitLogManager: Found 0 orphan tasks and 0 rescan nodes > > 2014-02-24 10:11:31,000 INFO [master:hadoop-master:60000] > > zookeeper.ZooKeeper: Initiating client connection, > > connectString=192.168.14.35:2181 sessionTimeout=90000 > > watcher=hconnection-0x4a867fad > > 2014-02-24 10:11:31,012 INFO > > [master:hadoop-master:60000-SendThread(hadoop-master.payoda.com:2181)] > > zookeeper.ClientCnxn: Opening socket connection to server > > hadoop-master.payoda.com/192.168.14.35:2181. Will not attempt to > > authenticate using SASL (Unable to locate a login configuration) > > 2014-02-24 10:11:31,617 INFO > > [master:hadoop-master:60000-SendThread(hadoop-master.payoda.com:2181)] > > zookeeper.ClientCnxn: Socket connection established to > > hadoop-master.payoda.com/192.168.14.35:2181, initiating session > > 2014-02-24 10:11:31,617 INFO [master:hadoop-master:60000] > > zookeeper.RecoverableZooKeeper: Process > > identifier=hconnection-0x4a867fad connecting to ZooKeeper > > ensemble=192.168.14.35:2181 > > 2014-02-24 10:11:31,620 INFO > > [master:hadoop-master:60000-SendThread(hadoop-master.payoda.com:2181)] > > zookeeper.ClientCnxn: Session establishment complete on server > > hadoop-master.payoda.com/192.168.14.35:2181, sessionid = > > 0x1446360aa4a0001, negotiated timeout = 90000 > > 2014-02-24 10:11:31,640 DEBUG [master:hadoop-master:60000] > > catalog.CatalogTracker: Starting catalog tracker > > org.apache.hadoop.hbase.catalog.CatalogTracker@3eaa3e5b > > **2014-02-24 10:11:31,684 FATAL [master:hadoop-master:60000] > > master.HMaster: Unhandled exception. Starting shutdown. > > java.lang.IllegalArgumentException: .META. no longer exists. The table > > has been renamed to hbase:meta** > > at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:292) > > at > > > org.apache.hadoop.hbase.zookeeper.ZKTable.populateTableStates(ZKTable.java:82) > > at org.apache.hadoop.hbase.zookeeper.ZKTable.(ZKTable.java:69) > > at > > > org.apache.hadoop.hbase.master.AssignmentManager.(AssignmentManager.java:281) > > at > > > org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:677) > > at > > > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:809) > > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:603) > > at java.lang.Thread.run(Thread.java:662) > > 2014-02-24 10:11:31,684 INFO [master:hadoop-master:60000] > > master.HMaster: Aborting > > 2014-02-24 10:11:31,711 DEBUG [master:hadoop-master:60000] > > catalog.CatalogTracker: Stopping catalog tracker > > org.apache.hadoop.hbase.catalog.CatalogTracker@3eaa3e5b > > 2014-02-24 10:11:31,712 DEBUG [master:hadoop-master:60000] > > master.HMaster: Stopping service threads > > 2014-02-24 10:11:31,712 INFO [master:hadoop-master:60000] > > ipc.RpcServer: Stopping server on 60000 > > 2014-02-24 10:11:31,712 INFO [RpcServer.handler=15,port=60000] > > ipc.RpcServer: RpcServer.handler=15,port=60000: exiting > > 2014-02-24 10:11:31,712 INFO [RpcServer.handler=23,port=60000] > > ipc.RpcServer: RpcServer.handler=23,port=60000: exiting > > 2014-02-24 10:11:32,129 INFO [master:hadoop-master:60000] > > master.HMaster: Stopping infoServer > > 2014-02-24 10:11:32,138 INFO [RpcServer.responder] ipc.RpcServer: > > RpcServer.responder: stopped > > 2014-02-24 10:11:32,138 INFO [RpcServer.responder] ipc.RpcServer: > > RpcServer.responder: stopping > > 2014-02-24 10:11:32,304 INFO > > [hadoop-master.payoda.com > > ,60000,1393236677609.splitLogManagerTimeoutMonitor] > > master.SplitLogManager$TimeoutMonitor: > > hadoop-master.payoda.com > ,60000,1393236677609.splitLogManagerTimeoutMonitor > > exiting > > 2014-02-24 10:11:32,304 INFO [RpcServer.listener,port=60000] > > ipc.RpcServer: RpcServer.listener,port=60000: stopping > > 2014-02-24 10:11:32,304 INFO > > [Replication.RpcServer.handler=2,port=60000] ipc.RpcServer: > > Replication.RpcServer.handler=2,port=60000: exiting > > 2014-02-24 10:11:32,304 INFO > > [Replication.RpcServer.handler=1,port=60000] ipc.RpcServer: > > Replication.RpcServer.handler=1,port=60000: exiting > > 2014-02-24 10:11:32,304 INFO > > [Replication.RpcServer.handler=0,port=60000] ipc.RpcServer: > > Replication.RpcServer.handler=0,port=60000: exiting > > 2014-02-24 10:11:32,304 INFO [RpcServer.handler=29,port=60000] > > ipc.RpcServer: RpcServer.handler=29,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=28,port=60000] > > ipc.RpcServer: RpcServer.handler=28,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=27,port=60000] > > ipc.RpcServer: RpcServer.handler=27,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=26,port=60000] > > ipc.RpcServer: RpcServer.handler=26,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=25,port=60000] > > ipc.RpcServer: RpcServer.handler=25,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=24,port=60000] > > ipc.RpcServer: RpcServer.handler=24,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=22,port=60000] > > ipc.RpcServer: RpcServer.handler=22,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=21,port=60000] > > ipc.RpcServer: RpcServer.handler=21,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=20,port=60000] > > ipc.RpcServer: RpcServer.handler=20,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=19,port=60000] > > ipc.RpcServer: RpcServer.handler=19,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=18,port=60000] > > ipc.RpcServer: RpcServer.handler=18,port=60000: exiting > > 2014-02-24 10:11:32,305 INFO [RpcServer.handler=17,port=60000] > > ipc.RpcServer: RpcServer.handler=17,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=16,port=60000] > > ipc.RpcServer: RpcServer.handler=16,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=14,port=60000] > > ipc.RpcServer: RpcServer.handler=14,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=13,port=60000] > > ipc.RpcServer: RpcServer.handler=13,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=12,port=60000] > > ipc.RpcServer: RpcServer.handler=12,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=11,port=60000] > > ipc.RpcServer: RpcServer.handler=11,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=10,port=60000] > > ipc.RpcServer: RpcServer.handler=10,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=9,port=60000] > > ipc.RpcServer: RpcServer.handler=9,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=8,port=60000] > > ipc.RpcServer: RpcServer.handler=8,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=7,port=60000] > > ipc.RpcServer: RpcServer.handler=7,port=60000: exiting > > 2014-02-24 10:11:32,306 INFO [RpcServer.handler=6,port=60000] > > ipc.RpcServer: RpcServer.handler=6,port=60000: exiting > > 2014-02-24 10:11:32,307 INFO [RpcServer.handler=5,port=60000] > > ipc.RpcServer: RpcServer.handler=5,port=60000: exiting > > 2014-02-24 10:11:32,307 INFO [RpcServer.handler=4,port=60000] > > ipc.RpcServer: RpcServer.handler=4,port=60000: exiting > > 2014-02-24 10:11:32,307 INFO [RpcServer.handler=3,port=60000] > > ipc.RpcServer: RpcServer.handler=3,port=60000: exiting > > 2014-02-24 10:11:32,307 INFO [RpcServer.handler=2,port=60000] > > ipc.RpcServer: RpcServer.handler=2,port=60000: exiting > > 2014-02-24 10:11:32,307 INFO [RpcServer.handler=1,port=60000] > > ipc.RpcServer: RpcServer.handler=1,port=60000: exiting > > 2014-02-24 10:11:32,307 INFO [RpcServer.handler=0,port=60000] > > ipc.RpcServer: RpcServer.handler=0,port=60000: exiting > > 2014-02-24 10:11:32,930 INFO [master:hadoop-master:60000] > > mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010 > > 2014-02-24 10:11:32,945 INFO [master:hadoop-master:60000] > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper > > sessionid=0x1446360aa4a0001 > > 2014-02-24 10:11:32,948 INFO [master:hadoop-master:60000] > > zookeeper.ZooKeeper: Session: 0x1446360aa4a0001 closed > > 2014-02-24 10:11:32,949 INFO [master:hadoop-master:60000-EventThread] > > zookeeper.ClientCnxn: EventThread shut down > > 2014-02-24 10:11:32,954 INFO [master:hadoop-master:60000] > > zookeeper.ZooKeeper: Session: 0x1446360aa4a0000 closed > > 2014-02-24 10:11:32,954 INFO [master:hadoop-master:60000] > > master.HMaster: HMaster main thread exiting > > 2014-02-24 10:11:32,955 INFO [main-EventThread] zookeeper.ClientCnxn: > > EventThread shut down > > 2014-02-24 10:11:32,955 ERROR [main] master.HMasterCommandLine: Master > > exiting > > **java.lang.RuntimeException: HMaster Aborted** > > at > > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192) > > at > > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2787) > > > > > > -- > > Dislike trivia, appreciate no-nonsense! > > > -- Dislike trivia, appreciate no-nonsense! --001a1135e1dc02f28304f3d7f58a--