hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-1098) IllegalStateException: Cannot set a region to be closed it it was not already marked as closing
Date Mon, 29 Dec 2008 23:57:44 GMT

     [ https://issues.apache.org/jira/browse/HBASE-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-1098:
-------------------------

    Attachment: 1098.patch

Make state members of RegionState volatile.

> IllegalStateException: Cannot set a region to be closed it it was not already marked
as closing
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1098
>                 URL: https://issues.apache.org/jira/browse/HBASE-1098
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: apurtel cluster
>            Reporter: stack
>             Fix For: 0.19.0
>
>         Attachments: 1098.patch
>
>
> content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 is problem region.  below is snippet
from master log.  Looks to me like things are proceeding in order.
> {code}
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server XX.XX.XX.51:60020
is overloaded. Server load: 74 avg: 48.0, slop: 0.1
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing
to reassign 26 regions. mostLoadedRegions has 10 regions in it.
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,07730cecbb5b13364cbe17af5778f983,1230448573272
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,29be0a07dece4e9bb318c4d253807287,1230356680860
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,23893c82d792ff5c9d3ff99123bc1773,1230390013767
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,e76475012b59500d25000aa7658bc279,1230498762282
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region result_domain,cn.com.sg.www,1230498785950
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
> 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to
close region content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678
> 2008-12-29 12:19:32,074 INFO org.apache.hadoop.hbase.master.RegionManager: Skipped 0
region(s) that are in transition states
> 2008-12-29 12:19:32,076 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: PendingOpenOperation from XX.XX.XX.51:60020
> 2008-12-29 12:19:32,076 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,42b19221f3a60945595b18017d6bacfa,1230357228381
open on XX.XX.XX.51:60020
> 2008-12-29 12:19:32,076 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row content,42b19221f3a60945595b18017d6bacfa,1230357228381 in region .META.,,1 with startcode
1230571104777 and server XX.XX.XX.51:60020
> 2008-12-29 12:19:32,077 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: PendingOpenOperation from XX.XX.XX.51:60020
> 2008-12-29 12:19:32,077 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241
open on XX.XX.XX.51:60020
> 2008-12-29 12:19:32,077 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 in region .META.,,1 with startcode
1230571104777 and server XX.XX.XX.51:60020
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,36946541ed9a62f419cf7238d32a6a38,1230448587552, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,94b170cae87992c33c3913d74bb257e4,1230498360162, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,8ac5fbdd63b313ce252838a189c73438,1230530000480, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,4b30e2a8b7949ed62e097824969f8bca,1230448501152, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,0c4b7c2825f674510a511233e9e0c539,1230530122456, false
> 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831, false
> 2008-12-29 12:19:32,299 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,300 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,94b170cae87992c33c3913d74bb257e4,1230498360162 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,8ac5fbdd63b313ce252838a189c73438,1230530000480 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,36946541ed9a62f419cf7238d32a6a38,1230448587552 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,4b30e2a8b7949ed62e097824969f8bca,1230448501152 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:32,304 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region content,0c4b7c2825f674510a511233e9e0c539,1230530122456 to server XX.XX.XX.34:60020
> 2008-12-29 12:19:35,080 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,07730cecbb5b13364cbe17af5778f983,1230448573272 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,07730cecbb5b13364cbe17af5778f983,1230448573272, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,29be0a07dece4e9bb318c4d253807287,1230356680860 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,23893c82d792ff5c9d3ff99123bc1773,1230390013767 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,29be0a07dece4e9bb318c4d253807287,1230356680860, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,e76475012b59500d25000aa7658bc279,1230498762282 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,23893c82d792ff5c9d3ff99123bc1773,1230390013767, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
result_domain,cn.com.sg.www,1230498785950 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,e76475012b59500d25000aa7658bc279,1230498762282, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of result_domain,cn.com.sg.www,1230498785950, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914, false
> 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678 from XX.XX.XX.51:60020
> 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322, false
> 2008-12-29 12:19:35,082 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing
loop: ProcessRegionClose of content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678, false
> 2008-12-29 12:19:35,083 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9
on 60000, call regionServerReport(address: XX.XX.XX.51:60020, startcode: 1230571104777, load:
(requests=0, regions=64, usedHeap=99, maxHeap=1777), [Lorg.
> apache.hadoop.hbase.HMsg;@3e2dce4e, [Lorg.apache.hadoop.hbase.HRegionInfo;@314382c6)
from XX.XX.XX.51:55371: error: java.io.IOException: java.lang.IllegalStateException: Cannot
set a region to be closed if it was not already marked as clo
> sing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
> java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed
if it was not already marked as closing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
>         at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1108)
>         at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:816)
>         at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:377)
>         at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:327)
>         at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:240)
>         at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:570)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message