From dev-return-4216-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Sat Jul 17 14:58:12 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 9D76A180643 for ; Sat, 17 Jul 2021 16:58:12 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 58B5A607E9 for ; Sat, 17 Jul 2021 14:58:10 +0000 (UTC) Received: (qmail 27863 invoked by uid 500); 17 Jul 2021 14:58:10 -0000 Mailing-List: contact dev-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list dev@hudi.apache.org Received: (qmail 27851 invoked by uid 99); 17 Jul 2021 14:58:09 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2021 14:58:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id F02FD1FF489 for ; Sat, 17 Jul 2021 14:58:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -0.002 X-Spam-Level: X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id w8XPqtvrzdPX for ; Sat, 17 Jul 2021 14:58:06 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.177; helo=mail-lj1-f177.google.com; envelope-from=vinay18.patil@gmail.com; receiver= Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 28918BC48D for ; Sat, 17 Jul 2021 14:58:06 +0000 (UTC) Received: by mail-lj1-f177.google.com with SMTP id bn5so17925992ljb.10 for ; Sat, 17 Jul 2021 07:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=jjoBSkBzNUSk7NGEfDflmiozHy6tIFw5qXaq5GCIzw8=; b=geFCiZAIwRNgh7G12MZh/78c6sdMbPpUfWZ59DowdCRhSMmbnKC1TrRPQbr+OoVGLG 3n/ggq0kKbaf92UNYVlC4fvdYFD4UNX2e61PPg8Ap/JEo9wLkcycbxw3mI/kBOUIYJkW K+Y67YHIvA1p+r81JYUCNSfOb/P+LUcL95/p+dDScbcRoIuZOriCEqNpq70HotYTBerQ qu5KRB1skokVaW+DBmWWMEkVLuq77lPbCeJbWVKrXpwnJYKhGMiDHBMUZXuR27rp9+Vz sLfgQToQZA9IFF5rC5AmenmUWDyjzojh7l0osfrjCM9EDSjxlRi+kfO3h3qEWpNKKdaH eTvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=jjoBSkBzNUSk7NGEfDflmiozHy6tIFw5qXaq5GCIzw8=; b=EKsTnLDRJzMu3fQLjBHUpin9h9KESkNe7hMiy0gkEq+8G0DylqK8/A52LN+fFbyCWg ME+5NhyA4tgJHkq3sip5D4hpY31XRBFeJHtEXa4rYqdFNj8SXCecWjVMn/f5abjg2ioK fvHORI00DIlPrKviAsPpGtpMlgwDhJuYGc9LJyC4w0r1FB9lNCqx5XhJ9TO4QQbyLAZK Rw5qes7wxweUwkut3cMuRI8CjvA8KoMgJ1f1xAmva/S/khrY7CZ3Ve10QUQgq6IgoTgo Pv89q2tKeWP5W2pasEnsZbUskh7fbIS7fjJXgpaTSm7WLE18S7C7awSc62CH3prheqOf f5Qw== X-Gm-Message-State: AOAM532qlMmxfyEy5eL6DAKLRamEqEE43I2GL6nm70kOjCnTUaaW5RjM pgJxAVCUSUO4UT2aqz/4TlI+JL4Kub/WRUwUv4o7Ms2Wy8q3YQ== X-Google-Smtp-Source: ABdhPJzS9iJXz+h8uLvJoSTOxUM5Hs39jjD8dnWZAeqqkJWOX15oYXFncVakWsfxd+fnX2LOPrZlTvlSLhyA8wyxhTE= X-Received: by 2002:a2e:8956:: with SMTP id b22mr8643725ljk.248.1626533884750; Sat, 17 Jul 2021 07:58:04 -0700 (PDT) MIME-Version: 1.0 From: Vinay Patil Date: Sat, 17 Jul 2021 20:27:53 +0530 Message-ID: Subject: [DISCUSS] Create Spark and Flink utilities module To: dev@hudi.apache.org Content-Type: multipart/alternative; boundary="0000000000006a6edb05c752ee0e" --0000000000006a6edb05c752ee0e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Team, As part of https://issues.apache.org/jira/browse/HUDI-1872, we are creating a separate flink-utilities module. Based on our discussion on the PR, should we even create a spark-utilities module. This would look like : hudi-utilities =E2=94=9C=E2=94=80=E2=94=80 hudi-flink-utilities =E2=94=94=E2=94=80=E2=94=80 hudi-spark-utilities This would also mean to create separate utilities-bundle for Flink and Spark, hudi-utilities-bundle =E2=94=9C=E2=94=80=E2=94=80 hudi-flink-utilities-bundle =E2=94=94=E2=94=80=E2=94=80 hudi-spark-utilities-bundle This is not a backward compatible change as users will have to provide an engine specific bundle. IMO, since Hudi is supporting Flink and Spark it will be good to have engine specific bundle. What do you think? Regards, Vinay Patil --0000000000006a6edb05c752ee0e--