From dev-return-4126-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Sun Jun 20 15:20:49 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 2595918064A for ; Sun, 20 Jun 2021 17:20:49 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 37D9C606DC for ; Sun, 20 Jun 2021 15:20:30 +0000 (UTC) Received: (qmail 64891 invoked by uid 500); 20 Jun 2021 15:20:29 -0000 Mailing-List: contact dev-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list dev@hudi.apache.org Received: (qmail 64863 invoked by uid 99); 20 Jun 2021 15:20:28 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jun 2021 15:20:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id F2ACC1FF4AE; Sun, 20 Jun 2021 15:20:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.248 X-Spam-Level: X-Spam-Status: No, score=0.248 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id lmB72ViifQu7; Sun, 20 Jun 2021 15:20:27 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.182; helo=mail-lj1-f182.google.com; envelope-from=leesf0315@gmail.com; receiver= Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 8F71BBC491; Sun, 20 Jun 2021 15:20:26 +0000 (UTC) Received: by mail-lj1-f182.google.com with SMTP id c11so21440638ljd.6; Sun, 20 Jun 2021 08:20:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=PUX00eigq42RHOMXrRQ+JB/beznVsa+87sBKs0PvBUs=; b=Fa55+UsWScg5FKYRKDSsZWEwEKfCgA6XJGtwWGbMda5oIG8aP6jhIDaRzF2DMfCj6+ Ox8/vCL7iEkqt7O3IxcYgR8Qod1vJ8Jg4TmR/FTfcjmH5hZ/FjbSCCyqiSDc+JdU8LcN ZzqpOi57e0vEe7FKmqaaFdaag1A2OYOi904Ok8ZkkbgVUqBWu1Yl2Gd3DouCtWuPFjAb y74iI9KKWgcySuMyojXJuyHgyZJvi8t8nfY/9PXaWii0QmUszSRBFnBc5xD0BqmfHsmh bAhCdXlrYMSzePPMZhtJr5H17TPJ+rfzloVmVYVJB3D/DXJJ+RoDpnRvLufjGhNjabi2 Z3qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=PUX00eigq42RHOMXrRQ+JB/beznVsa+87sBKs0PvBUs=; b=Pj9pwpC6BtG+zfPika4ESHvkkWQlVUOXOt7yDZJ/9ZrdyIp+AgbRSYT9yXqthZC8fP l40leIiX1A67xQmznc4ts1nN9BjD3J5gLL3fuIBkWHLdlx3GL1U6poLcpmNRVp/YkNvk Uv8SQcIuQsmYs7VL90hgv9Noz3qBHbmrdHD2St5H5Z9ZelFXu0TCkbt5gFbRoO1ajgxn 1CGWl765BP7d6zf7/iXzgL0XwSKF7WOTvGM4g632l93JEM285x7gzGZY8k4YYlNfs8Vy 3qn8fevPy1WTpXjTpLafTq5XCAcfSggXe1A+8f5uOHouhiZgkPp/uZjNLPDmDwzRij/l nfyg== X-Gm-Message-State: AOAM533H4rrLuWqxLwDumJID7J6I9loTnM1zjvFWZut98eZQ2fa4FC1l F6I3urQX/YBtprGxMHhfmV3cIWW90hMxaaVt7jv86hIPnuY= X-Google-Smtp-Source: ABdhPJw52uwxJhG5jisnOqu3c8cOBdDCs+zGcTMSoi3XXKpZiqWXHNGa/ekn9txOmKTcovYgXYF5yqjQn62nPL7g3ls= X-Received: by 2002:a2e:804a:: with SMTP id p10mr18239184ljg.428.1624202425200; Sun, 20 Jun 2021 08:20:25 -0700 (PDT) MIME-Version: 1.0 From: leesf Date: Sun, 20 Jun 2021 23:19:00 +0800 Message-ID: Subject: [ANNOUNCE] Hudi Community Update(2021-06-06 ~ 2021-06-20) To: dev Cc: users@hudi.apache.org Content-Type: multipart/alternative; boundary="00000000000098f40905c5341823" --00000000000098f40905c5341823 Content-Type: text/plain; charset="UTF-8" Dear community, Nice to share Hudi community bi-weekly updates for 2021-06-06 ~ 2021-06-20 with updates on features, bug fixes and tests. ======================================= Features [CLI] Add fetching latest schema to table command in hudi-cli [1] [Spark Integration] Added support for SqlFileBasedTransformer [2] [Flink Integration] add BootstrapFunction to support index bootstrap [3] [Spark Integration] Basic Implement Of Spark Sql Support For Hoodie [4] [Core] Support configure KeyGenerator by type [5] [Spark Integration] Added SqlSource to fetch data from any partitions for backfill use case [6] [Flink Integration] Support independent flink hudi compaction function [7] [Core] ORC reader writer Implementation [8] [Flink Integration] Support flink hive sync in batch mode [9] [Flink Integration] Add metadata cache to WriteProfile to reduce IO [10] [Flink Integration] Make flink writer as exactly-once by default [11] [Deltasteramer] Adds JDBC source support for DeltaStreamer [12] [1] https://issues.apache.org/jira/browse/HUDI-1914 [2] https://issues.apache.org/jira/browse/HUDI-1743 [3] https://issues.apache.org/jira/browse/HUDI-1924 [4] https://issues.apache.org/jira/browse/HUDI-1659 [5] https://issues.apache.org/jira/browse/HUDI-1929 [6] https://issues.apache.org/jira/browse/HUDI-1790 [7] https://issues.apache.org/jira/browse/HUDI-1984 [8] https://issues.apache.org/jira/browse/HUDI-765 [9] https://issues.apache.org/jira/browse/HUDI-2014 [10] https://issues.apache.org/jira/browse/HUDI-2030 [11] https://issues.apache.org/jira/browse/HUDI-2040 [12] https://issues.apache.org/jira/browse/HUDI-251 ======================================= Bugs [Spark Integration] Add Default value for HIVE_AUTO_CREATE_DATABASE_OPT_KEY in HoodieSparkSqlWriter [1] [Flink Integration] BucketAssignFunction use ValueState instead of MapState [2] [Flink Integration] Skip Commits with empty files [3] [Core] Fix NPE when avro field value is null [4] [Flink Integration] Skip creating marker files for flink merge handle [5] [Flink Integration] Fix non partition table hive meta sync for flink writer [6] [Flink Integration] Release the new records map for merge handle #close [7] [Flink Integration] Release the new records iterator for append handle #close [8] [Flink Integratoin] Release file writer for merge handle #close [9] [Spark Integration] Fixing drop dups exception in bulk insert row writer path [10] [Flink Integration] Refresh the base file view cache for WriteProfile [11] [Flink Integration] Release writer for append handle #close [12] [Code Cleanup] Avoid the raw type usage in some classes under hudi-utilities module [13] [Core] Fix the filter condition is missing in the judgment condition of compaction instance [14] [Flink Integration] Fix flink operator uid to allow multiple pipelines in one job [15] [Spaark Integration] Fix RO Tables Returning Snapshot Result [16] [Spark Integration] Set up the file system view storage config for singleton embedded server write config every time [17] [Flink Integration] Make keygen class and keygen type optional for FlinkStreamerConfig [18] [Spark Integration] ClassCastException Throw When PreCombineField Is String Type [19] [Flink Integration] Move the compaction plan scheduling out of flink writer coordinator [20] [1] https://issues.apache.org/jira/browse/HUDI-1942 [2] https://issues.apache.org/jira/browse/HUDI-1931 [3] https://issues.apache.org/jira/browse/HUDI-1909 [4] https://issues.apache.org/jira/browse/HUDI-1895 [5] https://issues.apache.org/jira/browse/HUDI-1723 [6] https://issues.apache.org/jira/browse/HUDI-1987 [7] https://issues.apache.org/jira/browse/HUDI-1992 [8] https://issues.apache.org/jira/browse/HUDI-1994 [9] https://issues.apache.org/jira/browse/HUDI-2000 [10] https://issues.apache.org/jira/browse/HUDI-1991 [11] https://issues.apache.org/jira/browse/HUDI-1999 [12] https://issues.apache.org/jira/browse/HUDI-2022 [13] https://issues.apache.org/jira/browse/HUDI-2008 [14] https://issues.apache.org/jira/browse/HUDI-1955 [15] https://issues.apache.org/jira/browse/HUDI-2015 [16] https://issues.apache.org/jira/browse/HUDI-1879 [17] https://issues.apache.org/jira/browse/HUDI-2019 [18] https://issues.apache.org/jira/browse/HUDI-2032 [19] https://issues.apache.org/jira/browse/HUDI-2033 [20] https://issues.apache.org/jira/browse/HUDI-2036 ====================================== Tests [Tests] Move TestHiveMetastoreBasedLockProvider to functional [1] [Tests] Move CheckpointUtils test cases to independant class [2] [Tests] Fix Azure CI failure in TestParquetUtils [3] [1] https://issues.apache.org/jira/browse/HUDI-1950 [2] https://issues.apache.org/jira/browse/HUDI-2004 [3] https://issues.apache.org/jira/browse/HUDI-1950 Best, Leesf --00000000000098f40905c5341823--