hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HBASE-804) A region split that occurs with live scanners breaks them
Date Fri, 08 Aug 2008 20:07:44 GMT

     [ https://issues.apache.org/jira/browse/HBASE-804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jean-Daniel Cryans resolved HBASE-804.
--------------------------------------

    Resolution: Invalid
      Assignee:     (was: Jean-Daniel Cryans)

Ok, with a bit more of code reading both in HR and in the map task, I figured that it was
a misuse of the Map that provoked the error. Here's how it goes:

The Map opens a scanner on the region
In each call of the map method, it deleted/wrote stuff for the given row key in that same
row
Sometimes, a split occurred during that time on that region and it waits for all scanners
to finish/timeout
Since the map tries to write stuff and that it can't (because the split has a write lock),
the scanner timeouts
Finally, when the split is done, the write resumes and when the scanner tries to get the next
row, it fails.

> A region split that occurs with live scanners breaks them
> ---------------------------------------------------------
>
>                 Key: HBASE-804
>                 URL: https://issues.apache.org/jira/browse/HBASE-804
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.1.3, 0.2.0
>            Reporter: Jean-Daniel Cryans
>             Fix For: 0.2.1, 0.3.0
>
>
> In many failed map tasks we see things like this:
> {code}
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 64020, call next(-8330618580557781998)
from 192.168.1.84:59678: error: org.apache.hadoop.hbase.UnknownScannerException: Name: -8330618580557781998
> {code}
> Here is an excerpt of the regionserver log:
> {code}
> 2008-08-08 10:49:13,276 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split
of region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
> 2008-08-08 10:49:13,310 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions
and cache flushes disabled for region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
> 2008-08-08 10:49:13,310 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates and
scanners disabled for region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
> 2008-08-08 10:49:13,310 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: waiting for
1 scanners to finish
> 2008-08-08 10:50:13,306 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
-8330618580557781998 lease expired
> 2008-08-08 10:50:13,307 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active
scanners for region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
> 2008-08-08 10:50:13,307 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row
locks outstanding on region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
> 2008-08-08 10:50:13,307 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272.
Current region memcache size 3.7m
> 2008-08-08 10:50:13,821 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/amsterdam_migration/entities/858230020/attribute/mapfiles/5987693228942080783
with 54576 entries, sequence id 9333266, data size 3.7m, file size 4.5m
> 2008-08-08 10:50:13,821 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished
memcache flush for region entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
in 514ms, sequence id=9333266, compaction requested=true
> 2008-08-08 10:50:13,821 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 858230020/attribute
> 2008-08-08 10:50:13,822 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 858230020/context
> 2008-08-08 10:50:13,822 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 858230020/view
> 2008-08-08 10:50:13,822 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed entities,North_America-United_States-Missouri-Oran-b9c7e44a-4a2b-11dd-abd4-1231390079f2,1218205066272
> ...
> 2008-08-08 10:50:23,437 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 64020,
call next(-8330618580557781998) from 192.168.1.84:59678: error: org.apache.hadoop.hbase.UnknownScannerException:
Name: -8330618580557781998
> org.apache.hadoop.hbase.UnknownScannerException: Name: -8330618580557781998
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1115)
>   at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> {code} 
> We have to have some way of telling the client code that it's scanner lease was canceled
because of a split so that it stays hidden from the user.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message