hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] garyli1019 commented on issue #1362: HUDI-644 Enable user to get checkpoint from previous commits in DeltaStreamer
Date Fri, 06 Mar 2020 18:52:45 GMT
garyli1019 commented on issue #1362: HUDI-644 Enable user to get checkpoint from previous commits
in DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1362#issuecomment-595908998
 
 
   > Let me catch up on this discussion and circle back.. :)
   > 
   > Just one high level question (apologies if its already answered above).
   > 
   > why can't we use the checkpoint reset flag, if one-time manual restarts are needed
for deltastreamer? is it because its hard to compute that?
   
   Right. I need a robust way to generate the checkpoint from kafka-connect-hdfs managed files
and kafka-connect itself sometimes having an issue to retrieve checkpoint when the Kafka partition
number was large. The mechanism is to scan every single file and get the latest checkpoint
of each Kafka partition. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message