hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From leesf <leesf0...@gmail.com>
Subject [ANNOUNCE] Hudi Community Update(2021-07-18 ~ 2021-08-01)
Date Sun, 01 Aug 2021 15:49:00 GMT
Dear community,

Nice to share Hudi community bi-weekly updates for 2021-07-18 ~ 2021-08-01
with updates on features, bug fixes and tests.


=======================================
Features

[Core] Adding support to disable meta columns with bulk insert operation [1]
[DeltaStreamer] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to
DeltaStreamer [2]
[Spark Integration] MergeInto Support Partial Update For COW [3]
[Hive Integration] DeltaStreamer kafka source supports consuming from
specified timestamp [4]
[Hive Integration] Adding support for HMS for running DDL queries in
hive-sync [5]
[Docs] Automate the generation of configs webpage as configs are added to
Hudi repo [6]
[Core] Adding virtual key support to COW table [7]
[Flink Integration] Add rateLimiter when Flink writes to hudi [8]
[Core] Integrate consumers with rocksDB and compression within External
Spillable Map [9]
[Flink Integration] Add option 'hive_sync.mode' for flink writer [10]
[Spark Integration] Explicit parallelism for flink bulk insert [11]
[Hive Integration] Support setting hive sync partition extractor class
based on flink configuration [12]


[1] https://issues.apache.org/jira/browse/HUDI-2161
[2] https://issues.apache.org/jira/browse/HUDI-1860
[3] https://issues.apache.org/jira/browse/HUDI-1884
[4] https://issues.apache.org/jira/browse/HUDI-1447
[5] https://issues.apache.org/jira/browse/HUDI-1848
[6] https://issues.apache.org/jira/browse/HUDI-1241
[7] https://issues.apache.org/jira/browse/HUDI-2176
[8] https://issues.apache.org/jira/browse/HUDI-2215
[9] https://issues.apache.org/jira/browse/HUDI-2044
[10] https://issues.apache.org/jira/browse/HUDI-2228
[11] https://issues.apache.org/jira/browse/HUDI-2241
[12] https://issues.apache.org/jira/browse/HUDI-2184


=======================================
Bugs

[Flink Integration] Remove state in BootstrapFunction [1]
[Flink Integration] Create new bucket when NewFileAssignState filled[2]
[Flink Integration] Clean and reset the bootstrap events for coordinator
when task failover [3]
[Code Cleanup] Clean up Multiple versions of scala libraries detected
Warning [4]
[Flink Integraion] Add marker files for flink writer [5]
[Spark Integration] Sync Hive Failed When Execute CTAS In Spark2 And Spark3
[6]
[Core] Fix checkpoint blocked because getLastPendingInstant() action after
than restoreWriteMetadata() action [7]
[Flink Integration] Rollback inflight compaction for flink writer [8]
[Spark Integration] MergeInto MOR Table May Result InCorrect Result [9]
[Spark Integration] Missing PrimaryKey In Hoodie Properties For CTAS Table
 [10]
[Core] residual temporary files after clustering are not cleaned up [11]
[Core] Fix NPE of HoodieConfig [12]
[Core] Fix no value present in incremental query on MOR [13]
[Spark Integration] Fix Alter Partitioned Table Failed [14]
[Flink Integration] Only sync hive meta on successful commit for flink
batch writer [15]
[Core] Make codahale times transient to avoid serializable exceptions [16]
[Core]] BucketAssigner generates the fileId evenly to avoid data skew [17]
[Hive Integration] Fix database alreadyExists exception while hive sync [18]
[Spark Integration] Performance loss with the additional
hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write [19]
[Spark Integration] Unpersist the input rdd after the commit is completed
to save the memory space for inline compaction [20]
[Spark Integration] Fix Exception Cause By Table Name Case Sensitivity For
Append Mode Write [21]
[Flink Integration] Default consumes from the latest instant for flink
streaming reader [22]
[Flink Integration] Builtin sort operator for flink bulk insert [23]
[Core] Fix missing HoodieWriteStat in HoodieCreateHandle [24]


[1] https://issues.apache.org/jira/browse/HUDI-2193
[2] https://issues.apache.org/jira/browse/HUDI-2145
[3] https://issues.apache.org/jira/browse/HUDI-2198
[4] https://issues.apache.org/jira/browse/HUDI-2192
[5] https://issues.apache.org/jira/browse/HUDI-2204
[6] https://issues.apache.org/jira/browse/HUDI-2195
[7] https://issues.apache.org/jira/browse/HUDI-2206
[8] https://issues.apache.org/jira/browse/HUDI-2205
[9] https://issues.apache.org/jira/browse/HUDI-2139
[10] https://issues.apache.org/jira/browse/HUDI-2212
[11] https://issues.apache.org/jira/browse/HUDI-2214
[12] https://issues.apache.org/jira/browse/HUDI-2219
[13] https://issues.apache.org/jira/browse/HUDI-2217
[14] https://issues.apache.org/jira/browse/HUDI-2223
[15] https://issues.apache.org/jira/browse/HUDI-2227
[16] https://issues.apache.org/jira/browse/HUDI-2240
[17] https://issues.apache.org/jira/browse/HUDI-2245
[18] https://issues.apache.org/jira/browse/HUDI-2244
[19] https://issues.apache.org/jira/browse/HUDI-1425
[20] https://issues.apache.org/jira/browse/HUDI-2117
[21] https://issues.apache.org/jira/browse/HUDI-2251
[22] https://issues.apache.org/jira/browse/HUDI-2252
[23] https://issues.apache.org/jira/browse/HUDI-2254
[24] https://issues.apache.org/jira/browse/HUDI-2218

======================================
Tests

[Tests] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert
node [1]
[Tests] Fix NullPointerException in TestHoodieConsoleMetrics [2]
[Tests] Refactoring few tests to reduce runningtime. DeltaStreamer and
MultiDeltaStreamer tests. Bulk insert row writer tests [3]

[1] https://issues.apache.org/jira/browse/HUDI-2007
[2] https://issues.apache.org/jira/browse/HUDI-2211
[3] https://issues.apache.org/jira/browse/HUDI-2253

Best,
Leesf

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message