hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanjia Gary Li (Jira)" <j...@apache.org>
Subject [jira] [Created] (HUDI-644) Enable to retrieve checkpoint from previous commits in Delta Streamer
Date Thu, 27 Feb 2020 06:46:00 GMT
Yanjia Gary Li created HUDI-644:
-----------------------------------

             Summary: Enable to retrieve checkpoint from previous commits in Delta Streamer
                 Key: HUDI-644
                 URL: https://issues.apache.org/jira/browse/HUDI-644
             Project: Apache Hudi (incubating)
          Issue Type: Improvement
          Components: DeltaStreamer
            Reporter: Yanjia Gary Li
            Assignee: Yanjia Gary Li


This ticket is to resolve the following problem:

The user is using a homebrew Spark data source to read new data and write to Hudi table

The user would like to migrate to Delta Streamer

But the Delta Streamer only checks the last commit metadata, if there is no checkpoint info,
then the Delta Streamer will use the default. For Kafka source, it is LATEST. 

The user would like to run the homebrew Spark data source reader and Delta Streamer in parallel
to prevent data loss, but the Spark data source writer will make commit without checkpoint
info, which will reset the delta streamer. 

So if we have an option to allow the user to retrieve the checkpoint from previous commits
instead of the latest commit would be helpful for the migration. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message