From dev-return-4295-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Sun Aug 15 01:12:25 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id B061A180654 for ; Sun, 15 Aug 2021 03:12:25 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id E222560839 for ; Sun, 15 Aug 2021 01:12:24 +0000 (UTC) Received: (qmail 11951 invoked by uid 500); 15 Aug 2021 01:12:23 -0000 Mailing-List: contact dev-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list dev@hudi.apache.org Received: (qmail 11939 invoked by uid 99); 15 Aug 2021 01:12:23 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2021 01:12:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id A56C11FF40D for ; Sun, 15 Aug 2021 01:12:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -0.002 X-Spam-Level: X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id K10pQDQ_47sZ for ; Sun, 15 Aug 2021 01:12:21 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.219.50; helo=mail-qv1-f50.google.com; envelope-from=n.siva.b@gmail.com; receiver= Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 21321BB8EA for ; Sun, 15 Aug 2021 01:12:21 +0000 (UTC) Received: by mail-qv1-f50.google.com with SMTP id v1so7392597qva.7 for ; Sat, 14 Aug 2021 18:12:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=7pdKh/q57XAE1LHBSYi8gAaEytQwD4KEHZ157gi9/Vk=; b=YQUlRjYTNxjsNZ0eYMqtR+HauF901/QIOyR3ohp1z0pmbaUXaNV+UrrwYqENZK8Out jTUNdEe3P1iqVD0uoOBxyUQLblxKg3dC6zt+/ZQHpIXO2xyPFQDBevAEEXyDdoAbaH7c ++KzeRgO10Fm3EnKzvQcgQm4nikHQH4cZhFniQKlSq1fc8uot87htv7WcOby7q9ke1z/ MxJ0NDnG7eakwXxOzgPIMTLuy66hC7D/iKnaPToFWVlqr/bLet0sL0Jwr19u48SSVw/s poUqfXNCFjDbOoXLuB1WoZ6IJa1eFiJnav/49sUBnGwr1K0Boq+/OAP3zbg5eFqOQF8b iWQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=7pdKh/q57XAE1LHBSYi8gAaEytQwD4KEHZ157gi9/Vk=; b=HpO0IYvcRSnvxOPQHzsxFsV2j6UCeC3nLLB5WEUagC0FC93qKWdbnW3PUVPyO39Lnj bjMNJSW8D8fiFCkG/idGyryxBXCQTV/l/a4r1w2XRPU2ltbyQmGIq8T/sV6VICjqYtd4 xnN1BNUIQtp4imHrm05DWhSMUXEr2SzAO/Iah5k5W90R803tv9W/oBdzu68skb2fiyYr Q3Q/qQNF0/cPmxZmWVWR9K+vzZ42Jui+fqySZBzRFfIOP86yK3uTF8Fde/onRTJeN8Ro Q3jIRrXWTOGPhLz1If/5hkvsIWiVn9+o3T4P2rDIu3nKoTfKJdgAMICz2WrH1TnerMMG P50A== X-Gm-Message-State: AOAM531LXEJGxK4CX33H+mI5/jMeo462Rs3rYBrB/imZ/otOP6zSaIyl RhasIeo8itvfI4pn3kOaObmUd5gTvG3UaVZxMEU8UoVznz0= X-Google-Smtp-Source: ABdhPJzUkPpyEjDaoCoQSjFpeMAt4y3/+d6lzZNalisb7/bOYhjOhaPw4+xUsdGGmB2bybrYfwYHNMjwL2/zJS1s1hA= X-Received: by 2002:a0c:f286:: with SMTP id k6mr9834165qvl.50.1628989934730; Sat, 14 Aug 2021 18:12:14 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Sivabalan Date: Sat, 14 Aug 2021 21:12:04 -0400 Message-ID: Subject: Re: [DISCUSS] Hudi 0.9.0 Release To: dev@hudi.apache.org Content-Type: multipart/alternative; boundary="00000000000067020905c98ec6ca" --00000000000067020905c98ec6ca Content-Type: text/plain; charset="UTF-8" Status update: all release blockers are landed. We are good to go ahead with RC work. On Fri, Aug 13, 2021 at 5:46 PM Udit Mehrotra wrote: > Hi Community, > > Here is a quick update on 0.9.0 release status. Over the last 10 days we > made significant progress on the release blockers previously mentioned in > the thread, thanks to all the owners. Here are the remaining blockers the > we are currently tracking: > > - [HUDI-2305] Add MARKERS.type and fix marker-based rollback > - [HUDI-2268] Add upgrade and downgrade to and from 0.9.0 > release-blockers > - [HUDI-2307] When using delete_partition with ds should not rely on the > primary key > - [HUDI-2151] Flipping defaults > - [HUDI-1897] Deltastreamer source for AWS S3 > - [HUDI-2120] [DOC] Update docs about schema in flink sql configuration > - [HUDI-2119] Ensure the rolled-back instance was previously synced to > the Metadata Table when syncing a Rollback Instant. > > We plan to resolve these soon and cut a RC by *tomorrow (August 14th, 2021) > end of day PST*. If you have any other blockers that you would like to > surface for Hudi 0.9.0, feel free to reach out. > > Thanks, > Udit > > On Fri, Aug 6, 2021 at 1:53 AM sagar sumit wrote: > > > Hi Udit, Vinoth > > > > End of next week sounds good. Apart from the issues listed, there is one > > more that we can take in this release: > > [HUDI-1897] DeltaStreamer Source for AWS S3 > > > > It's under review and should be closed by early next week. > > > > Regards, > > Sagar > > > > On 2021/08/06 00:55:19, Raymond Xu wrote: > > > +1 End of next week > > > > > > On Thu, Aug 5, 2021 at 3:06 PM Sivabalan wrote: > > > > > > > Yeah, end of next week sounds good. > > > > > > > > Here are the status updates wrt patches I am involved. > > > > > > > > Plan to get these in by early next week. > > > > - [HUDI-2208] Support Bulk Insert For Spark Sql (Owner: > pengzhiwei) > > > > - [HUDI-2250] Bulk insert support for tables w/ primary key > (Owner: > > > > Sivabalan) > > > > - [HUDI-1842] Spark Sql Support For The Exists Hoodie Table > (Owner: > > > > pengzhiwei) > > > > - [HUDI-1138] Re-implement marker files via timeline server > (Owner: > > > > Ethan Guo) > > > > - [HUDI-1129] Improving schema evolution support in hudi (Owner: > > > > Sivabalan) > > > > > > > > Mid next week: > > > > - [HUDI-2063] Add Doc For Spark Sql (DML and DDL) integration With > > Hudi > > > > (Owner: pengzhiwei) > > > > > > > > Waiting for reviews. Will try to get it in by early next week. If > we > > > > couldn't get this in, probably will skip this release. > > > > - [HUDI-1763] Fixing honoring of Ordering val in > > > > DefaultHoodieRecordPayload.preCombine (Owner: Sivabalan) > > > > > > > > Removed from release blockers: > > > > - [HUDI-1887] Setting default value to false for enabling schema > > post > > > > processor (Owner: Sivabalan) > > > > - [HUDI-1850] Fixing read of a empty table but with failed write > > (Owner: > > > > Sivabalan) > > > > > > > > > > > > On Thu, Aug 5, 2021 at 11:17 AM Vinoth Chandar > > wrote: > > > > > > > > > Any other thoughts? Love to lock this date down sooner than later. > > > > > > > > > > Thanks > > > > > Vinoth > > > > > > > > > > On Tue, Aug 3, 2021 at 11:35 PM Udit Mehrotra > > wrote: > > > > > > > > > > > Agreed Vinoth. End of next week seems reasonable as a hard > > deadline for > > > > > > cutting the RC. > > > > > > > > > > > > If anyone thinks otherwise or needs more time, feel free to chime > > in. > > > > > > > > > > > > On Tue, Aug 3, 2021 at 8:10 PM Vinoth Chandar > > > > > wrote: > > > > > > > > > > > > > Thanks Udit! I propose we set end of next week as a hard > > deadline for > > > > > > > cutting the RC. Any thoughts? > > > > > > > > > > > > > > A good amount of progress is being made on these blockers, I > > think. > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 3, 2021 at 5:13 PM Udit Mehrotra < > uditme@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > Hi Community, > > > > > > > > > > > > > > > > As we draw close to doing Hudi 0.9.0 release, I am happy to > > share a > > > > > > > summary > > > > > > > > of the key features/improvements that would be going in the > > release > > > > > and > > > > > > > the > > > > > > > > current blockers for everyone's visibility. > > > > > > > > > > > > > > > > *Highlights* > > > > > > > > > > > > > > > > - [HUDI-1729] Asynchronous Hive sync and commits cleaning > > for > > > > > Flink > > > > > > > > writer > > > > > > > > - [HUDI-1738] Detect and emit deleted records for Flink > MOR > > > > table > > > > > > > > streaming read > > > > > > > > - [HUDI-1867] Support streaming reads for Flink COW table > > > > > > > > - [HUDI-1908] Global index for flink writer > > > > > > > > - [HUDI-1788] Support Insert Overwrite with Flink Writer > > > > > > > > - [HUDI-2209] Bulk insert for flink writer > > > > > > > > - [HUDI-1591] Support querying using non-globbed paths for > > Hudi > > > > > > Spark > > > > > > > > DataSource queries > > > > > > > > - [HUDI-1591] Partition pruning support for read optimized > > > > queries > > > > > > via > > > > > > > > Hudi Spark DataSource > > > > > > > > - [HUDI-1415] Register Hudi Table as a Spark DataSource > > Table > > > > with > > > > > > > > metastore. Queries via Spark SQL will be routed through > Hudi > > > > > > > DataSource > > > > > > > > (instead of InputFormat), thus making it more performant > > due to > > > > > > > Spark's > > > > > > > > native/optimized readers > > > > > > > > - [HUDI-1591] Partition pruning support for snapshot > > queries via > > > > > > Hudi > > > > > > > > Spark DataSource > > > > > > > > - [HUDI-1658] DML and DDL support via Spark SQL > > > > > > > > - [HUDI-1790] Add SqlSource for DeltaStreamer to support > > > > backfill > > > > > > use > > > > > > > > cases: > > > > > > > > - [HUDI-251] Add JDBC Source support for DeltaStreamer > > > > > > > > - [HUDI-1910] Support Kafka based checkpointing for > > > > > > > HoodieDeltaStreamer > > > > > > > > - [HUDI-1371] Support metadata based listing for Spark > > > > DataSource > > > > > > and > > > > > > > > Spark SQL > > > > > > > > - [HUDI-2013] [HUDI-1717] [HUDI-2089] [HUDI-2016] > > Improvements > > > > to > > > > > > > > Metadata based listing > > > > > > > > - HUDI-89] Introduce a HoodieConfig/ConfigProperty > > framework to > > > > > > bring > > > > > > > > all configs under one roof > > > > > > > > - [HUDI-2124] Grafana dashboard for Hudi > > > > > > > > - [HUDI-1104] [HUDI-1105] [HUDI-2009] Improvements to Bulk > > > > Insert > > > > > > via > > > > > > > > row writing > > > > > > > > - [HUDI-1483] Async clustering for Delta Streamer > > > > > > > > - [HUDI-2235] Add virtual key support to Hudi > > > > > > > > - [HUDI-1848] Add support for Hive Metastore in > > Hive-sync-tool > > > > > > > > - In addition, there have been significant improvements > and > > bug > > > > > > fixes > > > > > > > to > > > > > > > > improve the overall stability of Flink Hudi integration > > > > > > > > > > > > > > > > *Current Blockers* > > > > > > > > > > > > > > > > - [HUDI-2208] Support Bulk Insert For Spark Sql (Owner: > > > > > pengzhiwei) > > > > > > > > - [HUDI-1256] Follow on improvements to HFile tables for > > > > metadata > > > > > > > based > > > > > > > > listing (Owner: None) > > > > > > > > - [HUDI-2063] Add Doc For Spark Sql (DML and DDL) > > integration > > > > With > > > > > > > Hudi > > > > > > > > (Owner: pengzhiwei) > > > > > > > > - [HUDI-1842] Spark Sql Support For The Exists Hoodie > Table > > > > > (Owner: > > > > > > > > pengzhiwei) > > > > > > > > - [HUDI-1138] Re-implement marker files via timeline > server > > > > > (Owner: > > > > > > > > Ethan Guo) > > > > > > > > - [HUDI-1985] Website redesign implementation (Owner: > Vinoth > > > > > > > > Govindarajan) > > > > > > > > - [HUDI-2232] MERGE INTO fails with table having nested > > struct > > > > > > (Owner: > > > > > > > > pengzhiwei) > > > > > > > > - [HUDI-1468] incremental read support with clustering > > (Owner: > > > > > > Liwei) > > > > > > > > - [HUDI-2250] Bulk insert support for tables w/ primary > key > > > > > (Owner: > > > > > > > > None) > > > > > > > > - [HUDI-2222] [SQL] Test catalog integration (Owner: Sagar > > > > Sumit) > > > > > > > > - [HUDI-2221] [SQL] Functionality testing with Spark 2 > > (Owner: > > > > > Sagar > > > > > > > > Sumit) > > > > > > > > - [HUDI-1887] Setting default value to false for enabling > > schema > > > > > > post > > > > > > > > processor (Owner: Sivabalan) > > > > > > > > - [HUDI-1850] Fixing read of a empty table but with failed > > write > > > > > > > (Owner: > > > > > > > > Sivabalan) > > > > > > > > - [HUDI-2151] Enable defaults for out of box performance > > (Owner: > > > > > > Udit > > > > > > > > Mehrotra) > > > > > > > > - [HUDI-2119] Ensure the rolled-back instance was > previously > > > > > synced > > > > > > to > > > > > > > > the Metadata Table when syncing a Rollback Instant (Owner: > > > > > Prashant > > > > > > > > Wason) > > > > > > > > - [HUDI-1458] Support custom clustering strategies and > > preserve > > > > > > commit > > > > > > > > time to support incremental read (Owner: Satish Kotha) > > > > > > > > - [HUDI-1763] Fixing honoring of Ordering val in > > > > > > > > DefaultHoodieRecordPayload.preCombine (Owner: Sivabalan) > > > > > > > > - [HUDI-1129] Improving schema evolution support in hudi > > (Owner: > > > > > > > > Sivabalan) > > > > > > > > - [HUDI-2120] [DOC] Update docs about schema in flink sql > > > > > > > configuration > > > > > > > > (Owner: Xianghu Wang) > > > > > > > > - [HUDI-2182] Support Compaction Command For Spark Sql > > (Owner: > > > > > > > > pengzhiwei) > > > > > > > > > > > > > > > > Please respond to the thread if you think that I have missed > > > > > capturing > > > > > > > any > > > > > > > > of the highlights or blockers for Hudi 0.9.0 release. For the > > > > owners > > > > > of > > > > > > > > these release blockers, can you please provide a specific > > timeline > > > > > you > > > > > > > are > > > > > > > > willing to commit to for finishing these so we can cut an RC > ? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Udit > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > -Sivabalan > > > > > > > > > > -- Regards, -Sivabalan --00000000000067020905c98ec6ca--