From users-return-57-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Sun Oct 11 15:14:30 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id A549A18065E for ; Sun, 11 Oct 2020 17:14:30 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id DACDE122466 for ; Sun, 11 Oct 2020 15:14:29 +0000 (UTC) Received: (qmail 57216 invoked by uid 500); 11 Oct 2020 15:14:29 -0000 Mailing-List: contact users-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@hudi.apache.org Delivered-To: mailing list users@hudi.apache.org Received: (qmail 57196 invoked by uid 99); 11 Oct 2020 15:14:28 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Oct 2020 15:14:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 4ABB21FF39C; Sun, 11 Oct 2020 15:14:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.249 X-Spam-Level: X-Spam-Status: No, score=0.249 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id bx9iRes2t-8T; Sun, 11 Oct 2020 15:14:27 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::d29; helo=mail-io1-xd29.google.com; envelope-from=leesf0315@gmail.com; receiver= Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 677707F9AD; Sun, 11 Oct 2020 15:14:27 +0000 (UTC) Received: by mail-io1-xd29.google.com with SMTP id q202so5821924iod.9; Sun, 11 Oct 2020 08:14:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=o+oTpy9vSISSLTt8AwNQZpiXXSd/Ei7cI0x/rUetWY4=; b=Qg7hbWP6AGgPfLpQX/Ag2LXM7En38wnToBaiyYMcCo+0mH4r4JdvtXJ2pkHVOj7fhz 3EFSijDVYFGTMjOOgJbU12+P4PDvN6ubBYGKeFMTes6IUmW9rlbFTem2ffHF9ZR12FqE 0VNZIFUL7/MF1J28YmohgZBBqOk/GcnYcpV+N7WQLtbUxFQpaNPPKt/oqMy2vHG2e4f7 /toEJF8qzMJ1ZQ4LlCWFy3SB4bdGJlrVbeERaxXIygwC/tmtM+242pm/UBiQJTltqeWr Zv7ym+kw6fIhs4mZEdBb0Hfz1ZO6QnLIkGKcQcNB6+OBgzHSzy+qjvTzNy7eTK+RVyXS 6Ttg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=o+oTpy9vSISSLTt8AwNQZpiXXSd/Ei7cI0x/rUetWY4=; b=FCQ5oQN+1fHzw6ZBIlfVdHJaIecMhW13TOc3/YWFw6Lbj9SJMIXwVvn+nkw53YXOjk DHjCTKvj5GFoGBUwVueb6uawfGLoUh5U1PB0OeSJyjEvfra5iDZOlp/bER7xnpJxBYs3 QdX35uWbLnfy3S3V1SMOgNAmTGWKf0N/X9Oc7zjzegf5nfAO/xZbyGZGxM13Sod2t4kW kZJ7OTyzis5ls6d8BXBnyz88jdQOqSB3/l+a5DiV0Jw03Kqe5k13lrrWoq0ciMdtB7rL i8q33QWs7HRibwIv8o929ZhriV3E+0hLE60nD5pA4j+65VogKapwairaX88Kz6zRw4Oh 2LUA== X-Gm-Message-State: AOAM531g6udkMzWfR1iDsJrjFNS4zyshEDIZ+00+76GYRSgoDaHCzCdU sUE29s0IOrR4DQiVWuwn8UbIh032NTFana8vcKBWeZ/Coee3gg== X-Google-Smtp-Source: ABdhPJySFFjydHZNKpPgYS/bpWqTViRU67OXHj/xslGP8WHky6BLbDnLrqm3BAgVTq/T0Kx0o3FMsqRgbkmyIAQAxpE= X-Received: by 2002:a6b:3e84:: with SMTP id l126mr14448907ioa.118.1602429265794; Sun, 11 Oct 2020 08:14:25 -0700 (PDT) MIME-Version: 1.0 From: leesf Date: Sun, 11 Oct 2020 23:14:00 +0800 Message-ID: Subject: [ANNOUNCE] Hudi Community Bi-Weekly Update(2020-09-27 ~ 2020-10-11) To: dev Cc: users@hudi.apache.org Content-Type: multipart/alternative; boundary="0000000000002a5e1605b166a3e8" --0000000000002a5e1605b166a3e8 Content-Type: text/plain; charset="UTF-8" Dear community, Nice to share Hudi community bi-weekly update for 2020-09-27 ~ 2020-10-11 with updates on features, bugfixs and tests. ======================================= Features [Hive Sync] Make create hive database automatically configurable [1] [Writer Core] Deltastreamer Kafka consumption delay reporting indicators [2] [Writer Core] Introduce REPLACE top level action. Implement insert_overwrite operation on top of replace action [3] [Writer Core] Refactor hudi-client to support multi-engine [4] [Metrics] Added an API to shutdown and remove the metrics reporter [5] [Writer Core] add port configuration for EmbeddedTimelineService [6] [Spark Integration] use spark INCREMENTAL mode query hudi dataset support schema version [7] ======================================= Bugs [Writer Core] Avoid blank file created by HoodieLogFormatWriter [8] [Writer Core] relocated jetty in hudi-utilities-bundle pom [9] [Writer Core] Ordering Field should be optional when precombine is turned off [10] [DeltaStreamer] DeltaStreamer can now fetch schema before every run in continuous mode [11] ====================================== Tests [Test] Some improvements for the HUDI Test Suite [12] [Test] Migrate HoodieTestUtils APIs to HoodieTestTable [13] [1] https://issues.apache.org/jira/browse/HUDI-1192 [2] https://issues.apache.org/jira/browse/HUDI-1233 [3] https://issues.apache.org/jira/browse/HUDI-1072 [4] https://issues.apache.org/jira/browse/HUDI-1089 [5] https://issues.apache.org/jira/browse/HUDI-1305 [6] https://issues.apache.org/jira/browse/HUDI-1203 [7] https://issues.apache.org/jira/browse/HUDI-1301 [8] https://issues.apache.org/jira/browse/HUDI-840 [9] https://issues.apache.org/jira/browse/HUDI-1199 [10] https://issues.apache.org/jira/browse/HUDI-1208 [11] https://issues.apache.org/jira/browse/HUDI-603 [12] https://issues.apache.org/jira/browse/HUDI-1303 [13] https://issues.apache.org/jira/browse/HUDI-995 Best, Leesf --0000000000002a5e1605b166a3e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Dear community,

Nice to share Hudi community bi-wee= kly update for 2020-09-27 ~ 2020-10-11 with updates on features, bugfixs an= d tests.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Features
[Hive Sync] Make create hive database automatically configurable [1]<= br>[Writer Core] Deltastreamer Kafka consumption delay reporting indicators= [2]
[Writer Core] Introduce REPLACE top level action. Implement insert_= overwrite operation on top of replace action [3]
[Writer Core] Refactor = hudi-client to support multi-engine [4]
[Metrics] Added an API to shutdo= wn and remove the metrics reporter [5]
[Writer Core] add port configurat= ion for EmbeddedTimelineService [6]
[Spark Integration] use spark INCREM= ENTAL mode query hudi dataset support schema version [7]



=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Bugs

[Writer Core] Avoid = blank file created by HoodieLogFormatWriter [8]
[Writer Core] relocated = jetty in hudi-utilities-bundle pom [9]
[Writer Core] =C2=A0Ordering Fiel= d should be optional when precombine is turned off [10]
[DeltaStreamer] = DeltaStreamer can now fetch schema before every run in continuous mode [11]=



=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Tests

[= Test] Some improvements for the HUDI Test Suite [12]
[Test] Migrate Hood= ieTestUtils APIs to HoodieTestTable [13]


[1] https://issues.apache.org/jira/bro= wse/HUDI-1192
[2] https://issues.apache.org/jira/browse/HUDI-1233
[3] https://issues.apache= .org/jira/browse/HUDI-1072
[4] https://issues.apache.org/jira/browse/HUDI-1089<= br>[5] https://= issues.apache.org/jira/browse/HUDI-1305
[6] https://issues.apache.org/jira/browse/H= UDI-1203
[7] https://issues.apache.org/jira/browse/HUDI-1301
[8] https://issues.apache.org/j= ira/browse/HUDI-840
[9] https://issues.apache.org/jira/browse/HUDI-1199
[10]= https://issues= .apache.org/jira/browse/HUDI-1208
[11] https://issues.apache.org/jira/browse/HUDI-60= 3
[12] h= ttps://issues.apache.org/jira/browse/HUDI-1303
[13] https://issues.apache.org/jira/b= rowse/HUDI-995


Best,
Leesf
--0000000000002a5e1605b166a3e8--