Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.161.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ZMjUMb3cbjl+xEEpJDGzDMDY/o3t8Fk4JvEeiw8qyhMYDRPepavi7KEPKDxTYy/5jK
         EVEO0NG7ZcLASBOp0L0NVvkU8bxNVgxkQcXRxUvnXM+/iyu5F47wcZ8UWklNg0EvRzKG
         ur7PVmfWe+xLaCG4Xtb1co2bupZpRdsO7afPQ=
MIME-Version: 1.0
Date: Fri, 14 Jan 2011 15:04:03 -0800
Message-ID: <AANLkTik_YjcWNZ8LKBUdaYGRDN95tSsdOpwN56w-QKht@mail.gmail.com>
Subject: YouAreDeadException
From: Ted Yu <yuzhihong@gmail.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a2034af0710499d6730f

--20cf3054a2034af0710499d6730f
Content-Type: text/plain; charset=ISO-8859-1

I ran 0.90 RC3 in dev cluster.

I saw the following in region server log:

Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
currently processing sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378 as
dead server
    at
org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:197)
    at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247)
    at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:648)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)

    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:753)
    at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
    at $Proxy0.regionServerReport(Unknown Source)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:702)
    ... 2 more
2011-01-13 03:55:08,982 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
connectString=sjc1-hadoop0.sjc1.carrieriq.com:2181sessionTimeout=90000
watcher=hconnection
2011-01-13 03:55:08,914 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378,
load=(requests=0, regions=6, usedHeap=514, maxHeap=3983):
regionserver:60020-0x12d7b7b1c760004 regionserver:60020-0x12d7b7b1c760004
received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:328)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:246)
    at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)

---------------

And the following from master log:

2011-01-13 03:52:42,003 INFO
org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
ephemeral node deleted, processing expiration [
sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378]
2011-01-13 03:52:42,005 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Added=sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378 to dead servers,
submitted shutdown handler to be executed, root=false, meta=false
2011-01-13 03:52:42,005 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
for sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378
2011-01-13 03:52:42,092 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 1 hlog(s)
in hdfs://
sjc1-hadoop0.sjc1.carrieriq.com:9000/hbase/.logs/sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378
2011-01-13 03:52:42,093 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
Thread[WriterThread-0,5,main]: starting
2011-01-13 03:52:42,094 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
Thread[WriterThread-1,5,main]: starting
2011-01-13 03:52:42,096 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 1 of
1: hdfs://
sjc1-hadoop0.sjc1.carrieriq.com:9000/hbase/.logs/sjc1-hadoop1.sjc1.carrieriq.com,60020,1294856823378/sjc1-hadoop1.sjc1.carrieriq.com%3A60020.1294860449407,
length=0

Please advise what could be the cause.

Thanks

--20cf3054a2034af0710499d6730f--