hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool
Date Thu, 19 Mar 2020 17:44:03 GMT
vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper
tool
URL: https://github.com/apache/incubator-hudi/pull/1362#issuecomment-601321422
 
 
   Given that, do we still need the ability to search for the checkpoints in reverse time
order? tbh I don't see a value in it, since there cannot be multiple writers to a hudi table
anyway. 
   
   May be we can think about an `CheckPointProvider` abstraction where if DeltaStreamer cannot
find a checkpoint from the last delta commit/commit, it invokes `checkpointProvider.getCheckpoint()`.
We can actually introduce that in this PR and have two implementations
   
   1) (default)NoOpCheckpointProvider (throws an error if it cannot find a checkpoint) 
   2) ScanOlderCommitsCheckpointProvider (what you have now) 
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message