hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1243) oldlogfile.dat is screwed, so is it's region
Date Fri, 06 Mar 2009 21:05:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679722#action_12679722
] 

stack commented on HBASE-1243:
------------------------------

Hows this for a patch j-d?
{code}
Index: src/java/org/apache/hadoop/hbase/regionserver/HStore.java
===================================================================
--- src/java/org/apache/hadoop/hbase/regionserver/HStore.java   (revision 751066)
+++ src/java/org/apache/hadoop/hbase/regionserver/HStore.java   (working copy)
@@ -339,8 +339,14 @@
     TreeMap<HStoreKey, byte []> reconstructedCache =
       new TreeMap<HStoreKey, byte []>(new HStoreKey.HStoreKeyWritableComparator(this.info));
       
-    SequenceFile.Reader logReader = new SequenceFile.Reader(this.fs,
-        reconstructionLog, this.conf);
+    SequenceFile.Reader logReader = null;
+    try {
+      logReader = new SequenceFile.Reader(this.fs, reconstructionLog, this.conf);
+    } catch (IOException e) {
+      LOG.warn("Failed opening reconstruction log though check for null-size passed. " +
+        "POSSIBLE DATA LOSS!! Soldiering on", e);
+      return;
+    }
{code}


> oldlogfile.dat is screwed, so is it's region
> --------------------------------------------
>
>                 Key: HBASE-1243
>                 URL: https://issues.apache.org/jira/browse/HBASE-1243
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>            Reporter: Jean-Daniel Cryans
>            Priority: Blocker
>             Fix For: 0.19.1
>
>
> Getting this when a node dies (happens frequently lately):
> {code}
> 2009-03-05 04:15:03,251 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region web_pages,http://fortcollins.gaymonkey.com/,1235836722125 to server 192.168.1.106:62020
> 2009-03-05 04:15:15,263 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
web_pages,http://fortcollins.gaymonkey.com/,1235836722125: java.io.IOException: Could not
obtain block: blk_5568212401457404905_251597 file=/hbase/amsterdam_factory/web_pages/1263377107/oldlogfile.log
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
>         at java.io.DataInputStream.readFully(DataInputStream.java:178)
>         at java.io.DataInputStream.readFully(DataInputStream.java:152)
>         at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
>         at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
>         at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>         at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>         at org.apache.hadoop.hbase.regionserver.HStore.doReconstructionLog(HStore.java:342)
>         at org.apache.hadoop.hbase.regionserver.HStore.runReconstructionLog(HStore.java:297)
>         at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:237)
>         at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
>         at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
>         at java.lang.Thread.run(Thread.java:619)
>  from 192.168.1.106:62020
> 2009-03-05 04:15:18,266 INFO org.apache.hadoop.hbase.master.RegionManager: assigning
region web_pages,http://fortcollins.gaymonkey.com/,1235836722125 to server 192.168.1.106:62020
> 2009-03-05 04:15:30,150 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 192.168.1.106:62020}
> 2009-03-05 04:15:30,276 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
web_pages,http://fortcollins.gaymonkey.com/,1235836722125: java.io.IOException: Could not
obtain block: blk_5568212401457404905_251597 file=/hbase/amsterdam_factory/web_pages/1263377107/oldlogfile.log
> ...
> {code}
> It does not recover, I have to manually delete the file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message