hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-471) IllegalStateException thrown in client after region was split and deleted
Date Thu, 06 Mar 2008 04:06:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575555#action_12575555
] 

stack commented on HBASE-471:
-----------------------------

So, working theory is that its the order in which we update .META. on split.  We first edit
the parent marking it offline, etc.  Then we add the lower split and then the top split. 
If a client comes in while parent is offline but before the lower daughter split has been
added, they get ISE ("Region is offline").  If the client comes in after parent has been offlined
but after the lower daughter has been added BUT before the upper daughter has been added requesting
a row from the top half of the split, they get WRE.

Working on a test to prove my theory.  

If theory is correct, fix is reversing order in which we add regions.   If we add the top
half first and there is a request for a key from the top half, the client gets the newly added
region.  If a request for a key from lower half, they'll get the parent.  On trying to get
the key from the parent, will get a NSRE or closed.  Will retry. Meantime the lower half should
have been added.  Then its ok to offline the parent. 

> IllegalStateException thrown in client after region was split and deleted
> -------------------------------------------------------------------------
>
>                 Key: HBASE-471
>                 URL: https://issues.apache.org/jira/browse/HBASE-471
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.1.0
>         Environment: Linux Debian, HBase 0.16.0
>            Reporter: Lars George
>         Attachments: FixHBase428.java, hbase-master-log.tar.gz, logs.tar.gz
>
>
> For some reason a client sometimes fails to locate a row with a IllegalStateException
when the region was split and deleted. 
> > [2008-02-25 16:12:39,171] ERROR [http-80-Processor20] archive.MultilingualArchive
- getDocument: An error occurred.
> > java.lang.IllegalStateException: region offline: pdc-docs,US7039976_20060509,1203981958556
> >         at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:432)
> >         at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
> >         at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
> >         at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:102)
> >         at org.apache.hadoop.hbase.HTable.get(HTable.java:280)
> Tracing the region on the master shows this:
> > 2008-02-25 16:09:38,761 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REGION_SPLIT
:
> +pdc-docs,US7039976_20060509,1203981958556 from 192.168.105.21:60020
> > 2008-02-25 16:09:38,761 INFO org.apache.hadoop.hbase.HMaster: region pdc-docs,US7039976_20060509,1203981958556
split. New regions
> +are: pdc-docs,US7039976_20060509,1203984578345, pdc-docs,US7046359_20060516,1203984578345
> > 2008-02-25 16:10:02,470 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner
regioninfo: {regionname:
> +pdc-docs,US7039976_20060509,1203981958556, startKey: <US7039976_20060509>, endKey:
<US7053021_20060530>, encodedName: 1260314009,
> +offline: true, split: true, tableDesc: {name: pdc-docs, families: {contents:={name:
contents, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}, language:={name: language,
max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}, mimetype:={name: mimetype,
max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}}}}, server: 192.168.105.21:60020,
startCode: 1203949130468
> > 2008-02-25 16:10:02,513 DEBUG org.apache.hadoop.hbase.HMaster: pdc-docs,US7039976_20060509,1203984578345
no longer has references
> +to pdc-docs,US7039976_20060509,1203981958556
> > 2008-02-25 16:10:02,516 DEBUG org.apache.hadoop.hbase.HMaster: pdc-docs,US7046359_20060516,1203984578345
no longer has references
> +to pdc-docs,US7039976_20060509,1203981958556
> > 2008-02-25 16:10:02,516 INFO org.apache.hadoop.hbase.HMaster: Deleting region pdc-docs,US7039976_20060509,1203981958556
because
> +daughter splits no longer hold references
> After discussion with st^ack it seems that the server is simply not retrying IllegalStateException
but IOExceptions only. Also see HBASE-452 which should be addressed at the same time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message