hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samir Ahmic (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
Date Wed, 15 Jun 2016 12:06:09 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331590#comment-15331590
] 

Samir Ahmic edited comment on HBASE-15908 at 6/15/16 12:05 PM:
---------------------------------------------------------------

I did not try other hadoop/hbase versions but i'm pretty sure that this will be issue whenever
hadoop native libs are loaded. I have checked hadoop-2.7.1 and hadoop-2.5.2 DataChecksum#verifyChunkedSums()
and in both branches there is this condition:
{code}
 if (NativeCrc32.isAvailable()) {
      NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
          fileName, basePos);
      return;
    }
{code} 

I was able to workaround this issue by making direct ByteBuffers and use them if native libs
are loaded. Here is diff:
{code}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
index a47cc12..48310c9 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
@@ -27,6 +27,7 @@ import org.apache.hadoop.fs.ChecksumException;
 import org.apache.hadoop.hbase.classification.InterfaceAudience;
 import org.apache.hadoop.hbase.util.ChecksumType;
 import org.apache.hadoop.util.DataChecksum;
+import org.apache.hadoop.util.NativeCodeLoader;
 
 /**
  * Utility methods to compute and validate checksums.
@@ -117,12 +118,27 @@ public class ChecksumUtil {
       ByteBuffer data = (ByteBuffer) buffer.duplicate().position(0).limit(onDiskDataSizeWithHeader);
       ByteBuffer checksums = (ByteBuffer) buffer.duplicate().position(onDiskDataSizeWithHeader)
           .limit(buffer.capacity());
+      if(NativeCodeLoader.isNativeCodeLoaded()) {
+        dataChecksum.verifyChunkedSums(directify(data), directify(checksums), pathName, 0);
+      } else {
       dataChecksum.verifyChunkedSums(data, checksums, pathName, 0);
+      }
     } catch (ChecksumException e) {
       return false;
     }
     return true;  // checksum is valid
   }
+  
+  private static ByteBuffer directify(ByteBuffer dataBuf) {
+    ByteBuffer newBuf = ByteBuffer.allocateDirect(dataBuf.capacity());
+    newBuf.position(dataBuf.position());
+    newBuf.mark();
+    newBuf.put(dataBuf);
+    newBuf.reset();
+    newBuf.limit(dataBuf.limit());
+    return newBuf;
+  }
+
 
   /**
    * Returns the number of bytes needed to store the checksums for
{code}


was (Author: asamir):
I did not try other hadoop/hbase versions but i'm pretty sure that this will be issue whenever
hadoop native libs are loaded. I have checked hadoop-2.7.1 and hadoop-2.5.2 DataChecksum#verifyChunkedSums()
and in both branches there is this condition:
{code}
 if (NativeCrc32.isAvailable()) {
      NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
          fileName, basePos);
      return;
    }
{code} 

I was able to workaround this issue by making direct ByteBuffers and use them if native libs
are loaded. I will attach diff. 

> Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
> ---------------------------------------------------------------------------------------
>
>                 Key: HBASE-15908
>                 URL: https://issues.apache.org/jira/browse/HBASE-15908
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 1.3.0
>            Reporter: Mikhail Antonov
>            Assignee: Mikhail Antonov
>            Priority: Blocker
>             Fix For: 1.3.0
>
>         Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum verification? I'm
seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file <file path>
> 	at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> 	at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> 	at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1135)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> 	at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> 	at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> 	at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> 	at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> 	at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> 	at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> 	at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> 	... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be direct buffers
> 	at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native Method)
> 	at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> 	at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> 	at org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:151)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV3.<init>(HFileReaderV3.java:78)
> 	at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:487)
> 	... 16 more
> Prior this change we won't use use native crc32 checksum verification as in Hadoop's
DataChecksum#verifyChunkedSums we would go this codepath
> if (data.hasArray() && checksums.hasArray()) {
>   <check native checksum, but using byte[] instead of byte buffers>
> }
> So we were fine. However, now we're dropping below and try to use the slightly different
variant of native crc32 (if one is available)  taking ByteBuffer instead of byte[], which
expects DirectByteBuffer, not Heap BB. 
> I think easiest fix working on all Hadoops would be to remove asReadonly() conversion
here:
> !validateChecksum(offset, onDiskBlockByteBuffer.asReadOnlyBuffer(), hdrSize)) {
> I don't see why do we need it. Let me test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message