hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sagar sumit <sagarsumi...@gmail.com>
Subject Re: [DISCUSS] Hudi 0.9.0 Release
Date Fri, 06 Aug 2021 08:53:11 GMT
Hi Udit, Vinoth

End of next week sounds good. Apart from the issues listed, there is one more that we can
take in this release: 
[HUDI-1897] DeltaStreamer Source for AWS S3

It's under review and should be closed by early next week.

Regards,
Sagar

On 2021/08/06 00:55:19, Raymond Xu <xu.shiyan.raymond@gmail.com> wrote: 
> +1 End of next week
> 
> On Thu, Aug 5, 2021 at 3:06 PM Sivabalan <n.siva.b@gmail.com> wrote:
> 
> > Yeah, end of next week sounds good.
> >
> > Here are the status updates wrt patches I am involved.
> >
> >   Plan to get these in by early next week.
> >    - [HUDI-2208] Support Bulk Insert For Spark Sql (Owner: pengzhiwei)
> >    - [HUDI-2250] Bulk insert support for tables w/ primary key (Owner:
> > Sivabalan)
> >    - [HUDI-1842] Spark Sql Support For The Exists Hoodie Table (Owner:
> >    pengzhiwei)
> >    - [HUDI-1138] Re-implement marker files via timeline server (Owner:
> >    Ethan Guo)
> >    - [HUDI-1129] Improving schema evolution support in hudi (Owner:
> >    Sivabalan)
> >
> >    Mid next week:
> >    - [HUDI-2063] Add Doc For Spark Sql (DML and DDL) integration With Hudi
> >    (Owner: pengzhiwei)
> >
> >   Waiting for reviews. Will try to get it in by early next week. If we
> > couldn't get this in, probably will skip this release.
> >    - [HUDI-1763] Fixing honoring of Ordering val in
> >    DefaultHoodieRecordPayload.preCombine (Owner: Sivabalan)
> >
> >    Removed from release blockers:
> >    - [HUDI-1887] Setting default value to false for enabling schema post
> >    processor (Owner: Sivabalan)
> >    - [HUDI-1850] Fixing read of a empty table but with failed write (Owner:
> >    Sivabalan)
> >
> >
> > On Thu, Aug 5, 2021 at 11:17 AM Vinoth Chandar <vinoth@apache.org> wrote:
> >
> > > Any other thoughts? Love to lock this date down sooner than later.
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Tue, Aug 3, 2021 at 11:35 PM Udit Mehrotra <uditme@apache.org> wrote:
> > >
> > > > Agreed Vinoth. End of next week seems reasonable as a hard deadline for
> > > > cutting the RC.
> > > >
> > > > If anyone thinks otherwise or needs more time, feel free to chime in.
> > > >
> > > > On Tue, Aug 3, 2021 at 8:10 PM Vinoth Chandar <vinoth@apache.org>
> > wrote:
> > > >
> > > > > Thanks Udit! I propose we set end of next week as a hard deadline
for
> > > > > cutting the RC. Any thoughts?
> > > > >
> > > > > A good amount of progress is being made on these blockers, I think.
> > > > >
> > > > >
> > > > > On Tue, Aug 3, 2021 at 5:13 PM Udit Mehrotra <uditme@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi Community,
> > > > > >
> > > > > > As we draw close to doing Hudi 0.9.0 release, I am happy to
share a
> > > > > summary
> > > > > > of the key features/improvements that would be going in the
release
> > > and
> > > > > the
> > > > > > current blockers for everyone's visibility.
> > > > > >
> > > > > > *Highlights*
> > > > > >
> > > > > >    - [HUDI-1729] Asynchronous Hive sync and commits cleaning
for
> > > Flink
> > > > > >    writer
> > > > > >    - [HUDI-1738] Detect and emit deleted records for Flink MOR
> > table
> > > > > >    streaming read
> > > > > >    - [HUDI-1867] Support streaming reads for Flink COW table
> > > > > >    - [HUDI-1908] Global index for flink writer
> > > > > >    - [HUDI-1788] Support Insert Overwrite with Flink Writer
> > > > > >    - [HUDI-2209] Bulk insert for flink writer
> > > > > >    - [HUDI-1591] Support querying using non-globbed paths for
Hudi
> > > > Spark
> > > > > >    DataSource queries
> > > > > >    - [HUDI-1591] Partition pruning support for read optimized
> > queries
> > > > via
> > > > > >    Hudi Spark DataSource
> > > > > >    - [HUDI-1415] Register Hudi Table as a Spark DataSource Table
> > with
> > > > > >    metastore. Queries via Spark SQL will be routed through Hudi
> > > > > DataSource
> > > > > >    (instead of InputFormat), thus making it more performant
due to
> > > > > Spark's
> > > > > >    native/optimized readers
> > > > > >    - [HUDI-1591] Partition pruning support for snapshot queries
via
> > > > Hudi
> > > > > >    Spark DataSource
> > > > > >    - [HUDI-1658] DML and DDL support via Spark SQL
> > > > > >    - [HUDI-1790] Add SqlSource for DeltaStreamer to support
> > backfill
> > > > use
> > > > > >    cases:
> > > > > >    - [HUDI-251] Add JDBC Source support for DeltaStreamer
> > > > > >    - [HUDI-1910] Support Kafka based checkpointing for
> > > > > HoodieDeltaStreamer
> > > > > >    - [HUDI-1371] Support metadata based listing for Spark
> > DataSource
> > > > and
> > > > > >    Spark SQL
> > > > > >    - [HUDI-2013] [HUDI-1717] [HUDI-2089] [HUDI-2016] Improvements
> > to
> > > > > >    Metadata based listing
> > > > > >    - HUDI-89] Introduce a HoodieConfig/ConfigProperty framework
to
> > > > bring
> > > > > >    all configs under one roof
> > > > > >    - [HUDI-2124] Grafana dashboard for Hudi
> > > > > >    - [HUDI-1104] [HUDI-1105] [HUDI-2009] Improvements to Bulk
> > Insert
> > > > via
> > > > > >    row writing
> > > > > >    - [HUDI-1483] Async clustering for Delta Streamer
> > > > > >    - [HUDI-2235] Add virtual key support to Hudi
> > > > > >    - [HUDI-1848] Add support for Hive Metastore in Hive-sync-tool
> > > > > >    - In addition, there have been significant improvements and
bug
> > > > fixes
> > > > > to
> > > > > >    improve the overall stability of Flink Hudi integration
> > > > > >
> > > > > > *Current Blockers*
> > > > > >
> > > > > >    - [HUDI-2208] Support Bulk Insert For Spark Sql (Owner:
> > > pengzhiwei)
> > > > > >    - [HUDI-1256] Follow on improvements to HFile tables for
> > metadata
> > > > > based
> > > > > >    listing (Owner: None)
> > > > > >    - [HUDI-2063] Add Doc For Spark Sql (DML and DDL) integration
> > With
> > > > > Hudi
> > > > > >    (Owner: pengzhiwei)
> > > > > >    - [HUDI-1842] Spark Sql Support For The Exists Hoodie Table
> > > (Owner:
> > > > > >    pengzhiwei)
> > > > > >    - [HUDI-1138] Re-implement marker files via timeline server
> > > (Owner:
> > > > > >    Ethan Guo)
> > > > > >    - [HUDI-1985] Website redesign implementation (Owner: Vinoth
> > > > > >    Govindarajan)
> > > > > >    - [HUDI-2232] MERGE INTO fails with table having nested struct
> > > > (Owner:
> > > > > >    pengzhiwei)
> > > > > >    - [HUDI-1468] incremental read support with clustering (Owner:
> > > > Liwei)
> > > > > >    - [HUDI-2250] Bulk insert support for tables w/ primary key
> > > (Owner:
> > > > > > None)
> > > > > >    - [HUDI-2222] [SQL] Test catalog integration (Owner: Sagar
> > Sumit)
> > > > > >    - [HUDI-2221] [SQL] Functionality testing with Spark 2 (Owner:
> > > Sagar
> > > > > >    Sumit)
> > > > > >    - [HUDI-1887] Setting default value to false for enabling
schema
> > > > post
> > > > > >    processor (Owner: Sivabalan)
> > > > > >    - [HUDI-1850] Fixing read of a empty table but with failed
write
> > > > > (Owner:
> > > > > >    Sivabalan)
> > > > > >    - [HUDI-2151] Enable defaults for out of box performance
(Owner:
> > > > Udit
> > > > > >    Mehrotra)
> > > > > >    - [HUDI-2119] Ensure the rolled-back instance was previously
> > > synced
> > > > to
> > > > > >    the Metadata Table when syncing a Rollback Instant (Owner:
> > > Prashant
> > > > > > Wason)
> > > > > >    - [HUDI-1458] Support custom clustering strategies and preserve
> > > > commit
> > > > > >    time to support incremental read (Owner: Satish Kotha)
> > > > > >    - [HUDI-1763] Fixing honoring of Ordering val in
> > > > > >    DefaultHoodieRecordPayload.preCombine (Owner: Sivabalan)
> > > > > >    - [HUDI-1129] Improving schema evolution support in hudi
(Owner:
> > > > > >    Sivabalan)
> > > > > >    - [HUDI-2120] [DOC] Update docs about schema in flink sql
> > > > > configuration
> > > > > >    (Owner: Xianghu Wang)
> > > > > >    - [HUDI-2182] Support Compaction Command For Spark Sql (Owner:
> > > > > >    pengzhiwei)
> > > > > >
> > > > > > Please respond to the thread if you think that I have missed
> > > capturing
> > > > > any
> > > > > > of the highlights or blockers for Hudi 0.9.0 release. For the
> > owners
> > > of
> > > > > > these release blockers, can you please provide a specific timeline
> > > you
> > > > > are
> > > > > > willing to commit to for finishing these so we can cut an RC
?
> > > > > >
> > > > > > Thanks,
> > > > > > Udit
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Regards,
> > -Sivabalan
> >
> 

Mime
View raw message