hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2364) when hbase regionserver restarts, it says "impossible state for createLease()"
Date Fri, 18 Jan 2008 02:06:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560195#action_12560195
] 

stack commented on HADOOP-2364:
-------------------------------

Here's more info...

Does above exceptions for 4 hours and then...

{code}
...
2008-01-16 07:18:58,757 DEBUG org.apache.hadoop.hbase.HRegionServer: Done telling master we
are up
2008-01-16 07:18:58,757 DEBUG org.apache.hadoop.hbase.HRegionServer: Config from master: fs.default.name=XX.XX.XX.XX:10000
2008-01-16 07:18:58,757 DEBUG org.apache.hadoop.hbase.HRegionServer: Config from master: hbase.rootdir=/hbase/XX.XX.XX.XX
...
{code}

... then it does this... (which don't look too pretty):

{code}
...
2008-01-16 07:18:59,517 INFO org.apache.hadoop.hbase.HRegionServer: HRegionServer started
at: XX.XX.XX.232:60020
2008-01-16 07:18:59,517 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020:
starting
2008-01-16 07:36:01,780 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:36:01,804 DEBUG org.apache.hadoop.hbase.HStore: starting .META.,,1/info (1028785192/info)
(no reconstruction log)
2008-01-16 07:36:01,832 DEBUG org.apache.hadoop.hbase.HStore: maximum sequence id for hstore
.META.,,1/info (1028785192/info) is 26067834
2008-01-16 07:36:01,852 DEBUG org.apache.hadoop.hbase.HRegion: Next sequence id for region
.META.,,1 is 26067835
2008-01-16 07:36:01,853 INFO org.apache.hadoop.hbase.HRegion: region .META.,,1 available
2008-01-16 07:36:01,854 DEBUG org.apache.hadoop.hbase.HLog: changing sequence number from
0 to 26067835
2008-01-16 07:37:04,137 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:37:10,144 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:37:21,476 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:39:01,468 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:39:22,489 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:40:27,559 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:12,603 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:27,619 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:36,628 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:42,633 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:48,640 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:57,650 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:42:03,657 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:42:21,675 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:43:03,719 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:46:47,721 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:05,738 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:11,744 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:43,403 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:49,408 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:48:04,423 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:48:37,457 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:52:18,805 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:52:30,817 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:55:39,002 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:59:40,590 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 07:59:55,605 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:03:34,072 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:03:40,079 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:01,099 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:10,108 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:16,113 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:22,120 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:46,724 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:52,730 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:05:16,754 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:05:22,760 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:09:11,033 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:12:01,093 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:12:07,100 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:13:55,994 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:14:05,002 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-01-16 08:17:20,146 WARN org.apache.hadoop.hbase.HRegionServer: Processing message (Retry:
0)
java.net.SocketTimeoutException: timed out waiting for rpc response
        at org.apache.hadoop.ipc.Client.call(Client.java:484)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
        at $Proxy0.regionServerReport(Unknown Source)
        at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:714)
        at java.lang.Thread.run(Unknown Source)
2008-01-16 08:17:20,147 FATAL org.apache.hadoop.hbase.HRegionServer: unable to report to master
for 63015 milliseconds - aborting server
..
{code}

.. and shuts itself down.

Back later w/ master side of things...

> when hbase regionserver restarts, it says "impossible state for createLease()"
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2364
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: Michael Bieniosek
>            Assignee: Jim Kellerman
>            Priority: Minor
>
> I restarted a regionserver, and got this error in its logs:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.AssertionError:
Impossible state for createLease(): Lease -435227488/-435227488 is still held.
>         at org.apache.hadoop.hbase.Leases.createLease(Leases.java:145)
>         at org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1278
> )
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>         at org.apache.hadoop.ipc.Client.call(Client.java:482)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
>         at $Proxy0.regionServerStartup(Unknown Source)
>         at org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.jav
> a:1025)
>         at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:659)
>         at java.lang.Thread.run(Unknown Source)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message