hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-534) Stores retaining references to long-deleted mapfiles
Date Mon, 24 Mar 2008 20:07:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581660#action_12581660
] 

stack commented on HBASE-534:
-----------------------------

Studying more, root of problem is double assignment of region.  Below is master log excerpts.
 See how pdc-docs,EP05011250NWA2,1205705554585 is doubly-assigned:

{code}
2008-03-16 15:12:36,626 DEBUG org.apache.hadoop.hbase.HMaster: Current assignment of pdc-docs,EP05011250NWA2,1205705554585
is not valid: storedInfo: null, startCode: -1, storedInfo.startCode: -1, unassignedRegions:
false, pendingRegions: false
2008-03-16 15:12:37,463 INFO org.apache.hadoop.hbase.HMaster: assigning region pdc-docs,EP05011250NWA2,1205705554585
to server 192.168.105.21:60020
2008-03-16 15:12:38,406 INFO org.apache.hadoop.hbase.HMaster: region pdc-docs,EP05011250NWA2,1205704064309
split. New regions are: pdc-docs,EP05011250NWA2,1205705554585, pdc-docs,EP05076594NWA2,1205705554586
2008-03-16 15:12:38,499 INFO org.apache.hadoop.hbase.HMaster: assigning region pdc-docs,EP05011250NWA2,1205705554585
to server 192.168.105.39:60020
2008-03-16 15:12:40,490 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN
: pdc-docs,EP05011250NWA2,1205705554585 from 192.168.105.21:60020
2008-03-16 15:12:40,490 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_OPEN :
pdc-docs,EP05011250NWA2,1205705554585 from 192.168.105.21:60020
2008-03-16 15:12:40,490 INFO org.apache.hadoop.hbase.HMaster: 192.168.105.21:60020 serving
pdc-docs,EP05011250NWA2,1205705554585
2008-03-16 15:12:40,491 INFO org.apache.hadoop.hbase.HMaster: regionname: pdc-docs,EP05011250NWA2,1205705554585,
startKey: <EP05011250NWA2>, endKey: <EP05076594NWA2>, encodedName: 1663054471,
tableDesc: {name: pdc-docs, families: {contents:={name: contents, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}, language:={name: language,
max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, mimetype:={name: mimetype, max versions: 3, compression: NONE, in memory: false, max
length: 2147483647, bloom filter: none}}} open on 192.168.105.21:60020
2008-03-16 15:12:40,491 INFO org.apache.hadoop.hbase.HMaster: updating row pdc-docs,EP05011250NWA2,1205705554585
in table .META.,,1 with startcode 1205660242301 and server 192.168.105.21:60020
2008-03-16 15:12:41,530 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN
: pdc-docs,EP05011250NWA2,1205705554585 from 192.168.105.39:60020
2008-03-16 15:12:41,530 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_OPEN :
pdc-docs,EP05011250NWA2,1205705554585 from 192.168.105.39:60020
2008-03-16 15:12:41,530 INFO org.apache.hadoop.hbase.HMaster: 192.168.105.39:60020 serving
pdc-docs,EP05011250NWA2,1205705554585
2008-03-16 15:12:41,530 INFO org.apache.hadoop.hbase.HMaster: regionname: pdc-docs,EP05011250NWA2,1205705554585,
startKey: <EP05011250NWA2>, endKey: <EP05076594NWA2>, encodedName: 1663054471,
tableDesc: {name: pdc-docs, families: {contents:={name: contents, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}, language:={name: language,
max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, mimetype:={name: mimetype, max versions: 3, compression: NONE, in memory: false, max
length: 2147483647, bloom filter: none}}} open on 192.168.105.39:60020
2006-03-16 15:12:41,530 INFO org.apache.hadoop.hbase.HMaster: updating row pdc-docs,EP05011250NWA2,1205705554585
in table .META.,,1 with startcode 1205660242518 and server 192.168.105.39:60020
2008-03-16 15:13:36,741 DEBUG org.apache.hadoop.hbase.HMaster: pdc-docs,EP05011250NWA2,1205705554585
no longer has references to pdc-docs,EP05011250NWA2,1205704064309
{code}

Happening because regionserver added regions to META, scan happened and marked the regions
as needing assignment because server was empty and startcode -1, regions were added to unassignedRegions
and actually assigned... then in came the SPLIT region message and regions were added again
to unassignedRegions (no check in split block if regions already present).

> Stores retaining references to long-deleted mapfiles
> ----------------------------------------------------
>
>                 Key: HBASE-534
>                 URL: https://issues.apache.org/jira/browse/HBASE-534
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.1.0
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.1.0
>
>
> Saw the following on the Lars clusters (He's up on 0.16.1 and very recent 0.1 branch)
trying to run a scan over all his content:
> {code}
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
>         at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2241)
>         at org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java:2362)
>         at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2152)
>         at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java:1640)
>         at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1214)
>         at org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1448)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:585)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> Caused by: java.io.FileNotFoundException: File hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281/contents/mapfiles/3435064940161142159/data
does not exist.
>         at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
>         at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1387)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1382)
>         at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:254)
>         at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
>         at org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:600)
>         at org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:655)
>         at org.apache.hadoop.hbase.HStoreFile$HalfMapFileReader.<init>(HStoreFile.java:758)
>         at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:424)
>         at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2216)
>         ... 11 more
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>         at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
>         at org.apache.hadoop.hbase.HTable$ClientScanner.nextScanner(HTable.java:874)
>         at org.apache.hadoop.hbase.HTable$ClientScanner.next(HTable.java:915)
>         at org.apache.hadoop.hbase.hql.SelectCommand.scanPrint(SelectCommand.java:233)
>         at org.apache.hadoop.hbase.hql.SelectCommand.execute(SelectCommand.java:100)
>         at org.apache.hadoop.hbase.hql.HQLClient.executeQuery(HQLClient.java:50)
>         at org.apache.hadoop.hbase.Shell.main(Shell.java:114)
> {code}
> The scanner breaks when it hits the above exception.  The odd thing is that the referenced
mapfile is out of a region that was deleted 4 days ago after purportedly all references had
been let go:
> {code}
> 2008-03-16 15:13:36,744 DEBUG org.apache.hadoop.hbase.HRegion: DELETING region hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message