Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1132797856.1217465193299.JavaMail.jira@brutus>
Date: Wed, 30 Jul 2008 17:46:33 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Commented: (HBASE-766) FileNotFoundException trying to load
 HStoreFile 'data'
In-Reply-To: <1618874412.1216836151674.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618569#action_12618569 ] 

Jonathan Gray commented on HBASE-766:
-------------------------------------

Stack,

I used the jar you built, confirmed across the cluster it was the one being used, and ran the test... I don't see any additional logging.  Check out the region logs on hb5 (where items table was located).

Here is the error again:

2008-07-30 17:41:58,970 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950. Current region memcache size 64.0m
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/items/1211238171/content/mapfiles/793072914512989834 with 323136 entries, sequence id 2527714, data size 64.0m, file size 68.8m
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 1454ms, sequence id=2527714, compaction requested=true
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region: items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:00,424 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:00,428 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Compaction size 396611811, skipped 1, 252131417
2008-07-30 17:42:00,454 DEBUG org.apache.hadoop.hbase.regionserver.HStore: started compaction of 2 files into /hbase/items/compaction.dir/1211238171/content/mapfiles/4240641699209031496
2008-07-30 17:42:08,256 DEBUG org.apache.hadoop.hbase.regionserver.HStore: moving /hbase/items/compaction.dir/1211238171/content/mapfiles/4240641699209031496 to /hbase/items/1211238171/content/mapfiles/8894612533024691980
2008-07-30 17:42:08,293 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 1211238171/content store size is 377.8m
2008-07-30 17:42:08,299 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 7sec
2008-07-30 17:42:08,301 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates and scanners disabled for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,305 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,305 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950. Current region memcache size 1.9m
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/items/1211238171/content/mapfiles/2539623802632323222 with 10032 entries, sequence id 2537747, data size 1.9m, file size 2.0m
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 101ms, sequence id=2537747, compaction requested=true
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/uclusters
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/covisit
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/content
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/clusters
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/readby
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/cfrecs
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/blooms
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/ranked
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/savedby
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/sentby
2008-07-30 17:42:08,408 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1211238171/receivedby
2008-07-30 17:42:08,408 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,638 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464928303/2132462761
2008-07-30 17:42:08,650 INFO org.apache.hadoop.hbase.regionserver.HStore: HSTORE_LOGINFOFILE 2132462761/readby/8482419619225016670-1211238171/6573863319629365608/bottom does not contain a sequence number - ignoring
2008-07-30 17:42:08,653 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
java.io.FileNotFoundException: File does not exist: hdfs://mb0:9000/hbase/items/1211238171/readby/mapfiles/6573863319629365608/data
        at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:444)
        at org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:437)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:219)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1618)
        at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:466)
        at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:405)
        at org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:800)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:133)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:86)


> FileNotFoundException trying to load HStoreFile 'data'
> ------------------------------------------------------
>
>                 Key: HBASE-766
>                 URL: https://issues.apache.org/jira/browse/HBASE-766
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 766.patch, hs.patch
>
>
> From Renaud Delbru up on the list:
> {code}
> Hi,
> after our issues ("Replay of HLog required", in a precious thread) with HBase, it seems that HBase has corrupted regions.
> We have, on the three region servers, errors stating that HBase cannot open certain regions because some map files on hdfs are missing (see the log attached).
> Do you have any ideas how to fix this ?
> Thanks.
> java.io.FileNotFoundException: File does not exist: hdfs://hadoop1.sindice.net:54310/hbase/page-repository/1105668475/field/mapfiles/5122893264992435570/data
>   at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
>   at org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:464)
>   at org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:409)
>   at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:236)
>   at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1575)
>   at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:451)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:901)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:876)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:816)
>   at java.lang.Thread.run(Thread.java:619)
> {code}
> I'd thought we'd added handling of this kind of event post HDFS crashings but looking in code, it seems that TRUNK may not have been fixed up properly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.