hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1396: [HUDI-687] Stop incremental reader on RO table before a pending compaction
Date Fri, 13 Mar 2020 06:01:07 GMT
vinothchandar commented on a change in pull request #1396: [HUDI-687] Stop incremental reader
on RO table before a pending compaction
URL: https://github.com/apache/incubator-hudi/pull/1396#discussion_r392043404
 
 

 ##########
 File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
 ##########
 @@ -116,6 +117,24 @@
     return returns.toArray(new FileStatus[returns.size()]);
   }
 
+  /**
+   * Filter any specific instants that we do not want to process.
+   * example timeline:
+   *
+   * t0 -> create bucket1.parquet
+   * t1 -> create and append updates bucket1.log
+   * t2 -> request compaction
+   * t3 -> create bucket2.parquet
+   *
+   * if compaction at t2 takes a long time, incremental readers on RO tables can move to
t3 and would skip updates in t1
+   *
+   * To workaround this problem, we want to stop returning data belonging to commits >
t2.
+   * After compaction is complete, incremental reader would see updates in t2, t3, so on.
+   */
+  protected HoodieTimeline filterInstantsTimeline(HoodieTimeline timeline) {
 
 Review comment:
   lets move the code from `HoodieDefaultTimeline` here? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message