hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-766) FileNotFoundException trying to load HStoreFile 'data'
Date Thu, 31 Jul 2008 00:46:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618569#action_12618569
] 

Jonathan Gray commented on HBASE-766:
-------------------------------------

Stack,

I used the jar you built, confirmed across the cluster it was the one being used, and ran
the test... I don't see any additional logging.  Check out the region logs on hb5 (where items
table was located).

Here is the error again:

2008-07-30 17:41:58,970 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950. Current region
memcache size 64.0m
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/items/1211238171/content/mapfiles/793072914512989834
with 323136 entries, sequence id 2527714, data size 64.0m, file size 68.8m
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 1454ms, sequence
id=2527714, compaction requested=true
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region: items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:00,424 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:00,428 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Compaction size
396611811, skipped 1, 252131417
2008-07-30 17:42:00,454 DEBUG org.apache.hadoop.hbase.regionserver.HStore: started compaction
of 2 files into /hbase/items/compaction.dir/1211238171/content/mapfiles/4240641699209031496
2008-07-30 17:42:08,256 DEBUG org.apache.hadoop.hbase.regionserver.HStore: moving /hbase/items/compaction.dir/1211238171/content/mapfiles/4240641699209031496
to /hbase/items/1211238171/content/mapfiles/8894612533024691980
2008-07-30 17:42:08,293 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed compaction
of 1211238171/content store size is 377.8m
2008-07-30 17:42:08,299 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 7sec
2008-07-30 17:42:08,301 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split
of region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and
cache flushes disabled for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates and scanners
disabled for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active
scanners for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,305 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,305 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950. Current region
memcache size 1.9m
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/items/1211238171/content/mapfiles/2539623802632323222
with 10032 entries, sequence id 2537747, data size 1.9m, file size 2.0m
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 101ms, sequence
id=2537747, compaction requested=true
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/uclusters
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/covisit
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/content
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/clusters
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/readby
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/cfrecs
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/blooms
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/ranked
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/savedby
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/sentby
2008-07-30 17:42:08,408 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/receivedby
2008-07-30 17:42:08,408 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,638 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464928303/2132462761
2008-07-30 17:42:08,650 INFO org.apache.hadoop.hbase.regionserver.HStore: HSTORE_LOGINFOFILE
2132462761/readby/8482419619225016670-1211238171/6573863319629365608/bottom does not contain
a sequence number - ignoring
2008-07-30 17:42:08,653 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
failed for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
java.io.FileNotFoundException: File does not exist: hdfs://mb0:9000/hbase/items/1211238171/readby/mapfiles/6573863319629365608/data
        at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:444)
        at org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:437)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:219)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1618)
        at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:466)
        at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:405)
        at org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:800)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:133)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:86)


> FileNotFoundException trying to load HStoreFile 'data'
> ------------------------------------------------------
>
>                 Key: HBASE-766
>                 URL: https://issues.apache.org/jira/browse/HBASE-766
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 766.patch, hs.patch
>
>
> From Renaud Delbru up on the list:
> {code}
> Hi,
> after our issues ("Replay of HLog required", in a precious thread) with HBase, it seems
that HBase has corrupted regions.
> We have, on the three region servers, errors stating that HBase cannot open certain regions
because some map files on hdfs are missing (see the log attached).
> Do you have any ideas how to fix this ?
> Thanks.
> java.io.FileNotFoundException: File does not exist: hdfs://hadoop1.sindice.net:54310/hbase/page-repository/1105668475/field/mapfiles/5122893264992435570/data
>   at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
>   at org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:464)
>   at org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:409)
>   at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:236)
>   at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1575)
>   at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:451)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:901)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:876)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:816)
>   at java.lang.Thread.run(Thread.java:619)
> {code}
> I'd thought we'd added handling of this kind of event post HDFS crashings but looking
in code, it seems that TRUNK may not have been fixed up properly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message