hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1332: [HUDI -409] Match header and footer block length to improve corrupted block detection
Date Tue, 03 Mar 2020 18:48:09 GMT
bvaradar commented on a change in pull request #1332: [HUDI -409] Match header and footer block
length to improve corrupted block detection
URL: https://github.com/apache/incubator-hudi/pull/1332#discussion_r387216621
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
 ##########
 @@ -239,12 +240,24 @@ private boolean isBlockCorrupt(int blocksize) throws IOException {
       return true;
     }
 
+    // check if the blocksize mentioned in the footer is the same as the header; by seeking
back the length of a long
+    // the backward seek does not incur additional IO as {@link org.apache.hadoop.hdfs.DFSInputStream#seek()}
+    // only moves the index. actual IO happens on the next read operation
+    inputStream.seek(inputStream.getPos() - Long.BYTES);
+    long blockSizeFromFooter = inputStream.readLong() - MAGIC_BUFFER.length;
 
 Review comment:
   Can you add the reason why we subtracting Magic header length

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message