hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pratyaksh Sharma (Jira)" <>
Subject [jira] [Commented] (HUDI-603) HoodieDeltaStreamer should periodically fetch table schema update
Date Sun, 16 Feb 2020 19:58:00 GMT


Pratyaksh Sharma commented on HUDI-603:

Should not be an issue if you are not using continuous mode. However this is a really good
feature to have in case of continuous mode in HoodieDeltaStreamer. Have you thought of some
approach to implement the same? [~yx3zhu]

> HoodieDeltaStreamer should periodically fetch table schema update
> -----------------------------------------------------------------
>                 Key: HUDI-603
>                 URL:
>             Project: Apache Hudi (incubating)
>          Issue Type: Bug
>          Components: DeltaStreamer
>            Reporter: Yixue Zhu
>            Priority: Major
>              Labels: evolution, schema
> HoodieDeltaStreamer create SchemaProvider instance and delegate to DeltaSync for periodical
sync. However, default implementation of SchemaProvider does not refresh schema, which can
change due to schema evolution. DeltaSync snapshot the schema when it creates writeClient,
using the SchemaProvider instance or pick up from source, and the schema for writeClient is
not refreshed during the loop of Sync.
> I think this needs to be addressed to support schema evolution fully.

This message was sent by Atlassian Jira

View raw message