From dev-return-4119-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Fri Jun 11 07:26:06 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id C16C2180654 for ; Fri, 11 Jun 2021 09:26:06 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 85F2A62909 for ; Fri, 11 Jun 2021 07:26:04 +0000 (UTC) Received: (qmail 28629 invoked by uid 500); 11 Jun 2021 07:26:01 -0000 Mailing-List: contact dev-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list dev@hudi.apache.org Received: (qmail 28556 invoked by uid 99); 11 Jun 2021 07:26:00 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2021 07:26:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 8FB411FF47B for ; Fri, 11 Jun 2021 07:25:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -0.002 X-Spam-Level: X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id TTDh2vKUMqFQ for ; Fri, 11 Jun 2021 07:25:59 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.173; helo=mail-lj1-f173.google.com; envelope-from=vinay18.patil@gmail.com; receiver= Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 78ECCBC48A for ; Fri, 11 Jun 2021 07:25:58 +0000 (UTC) Received: by mail-lj1-f173.google.com with SMTP id bn21so8369259ljb.1 for ; Fri, 11 Jun 2021 00:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Ite7ujSaSvvOxEzlaPIWEvEjeBPjTACRyRiehd3Wve8=; b=lR9n3KEYJQGaaLTph4DqpOqCZaDJnbJbA1roVBBKAuXiQSnjg7ntcRPGaBFs1DVPAa cZn1VtD1Yur3HYINHtkZPybusoK68yUAHoGC+PmLxugj0CSpuohwO6h4Io5NSWVfSOo8 tAaq9cxl7u9zAw2WYtKfDeayxNT/AWQn5ndiWLO0IcGZnAP0/vC1l4sA2XN4SRdrWcHn bbJn04WJzSNLrb4rM3mKmfnuAdiCNBq/C8gx9HUsIXNZanzIgXuWyfh7B1ymFByy/roO PfR0LE90M+o2iEfrzGNdoBHASvYcInavfnMRhvdMIJRvGNfKfolNpxaNtdcrVPreZT55 ghgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Ite7ujSaSvvOxEzlaPIWEvEjeBPjTACRyRiehd3Wve8=; b=lDmnN9rAOMJuqIfAyvl/CKyV4skOFNnprdvIgtcDPFMDOUD5wXPwNl/LM+vdDSLTQ/ eeDb53WpJep58h8gPd9SfklAWNxmoHCwiqOPTXroat8IpaqrVhD9uvXLWFBLwR8LgZ6J VZAfdfPBcYtpAIeBXgooMq3dsdtAl4pmoLtEMuR/+fJloqUwZwwDt75RhioTZAnpL7/O smQoQ27wUq/+4Lzf6g55+Bp6SVHWPazaRbJNOszitLWwranLuQRP1gOe6UGiNQ6sB+j3 G+UmFILS88JqHACQBbMk9Ben/hcRM312qoJh6pK5xRCaiQosJ0xkJS+Zm2bJR2Ydl7OE FRbQ== X-Gm-Message-State: AOAM532SQFec2mo8xX0v+Z/FlaptOxv+Q/1JlRkaIVT2NKDjPAY7LaS1 hodglS477mpzL+4M42X90QG6S7+f+dko6Kwy5Q1SCeau1QQ= X-Google-Smtp-Source: ABdhPJzhjQVpY6lZ6SGxmrajnKaUxxt6R5CSMoAzTnEL2I2UPq9MQqNlZCx6UWFFTGsv+XWv5LQNgdot0fSGE2bJTWI= X-Received: by 2002:a2e:9e1a:: with SMTP id e26mr1968044ljk.265.1623396357009; Fri, 11 Jun 2021 00:25:57 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vinay Patil Date: Fri, 11 Jun 2021 12:55:46 +0530 Message-ID: Subject: Re: [Discuss] Provide a Flag to choose between Flink or Spark To: dev@hudi.apache.org Content-Type: multipart/alternative; boundary="0000000000003073b305c4786bbf" --0000000000003073b305c4786bbf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you Danny for your response. Can we have a JIRA story where all the refactoring is required for Hudi-Flink code as well. I will create a task if we agree that a Flag will be helpful to choose different runners Regards, Vinay Patil On Fri, Jun 11, 2021 at 12:34 PM Danny Chan wrote: > Basically agree with that, but before that we may need some refactoring t= o > the existing code: > > Move the HoodieFlinkStreamer from the hudi-flink module into the > hudi-utilities to be together with the HoodieDeltaStreamer. > We are planning to add separate flink compaction programs too, which has > the same problem. > > Best, > Danny Chan > > Vinay Patil =E4=BA=8E2021=E5=B9=B46=E6=9C=889= =E6=97=A5=E5=91=A8=E4=B8=89 =E4=B8=8B=E5=8D=883:42=E5=86=99=E9=81=93=EF=BC= =9A > > > Hi Team, > > > > Currently, Hudi supports Flink as well Spark, there are two different > > classes > > 1. HoodieDeltaStreamer > > 2. FlinkHoodieDeltaStreamer > > > > Should we have a provision to pass the flag like --runner to choose > between > > Flink or Spark and have a single entry point class which will take all > the > > common configs. > > > > Based on the runner flag, we can call HoodieDeltaStreamer or > > FlinkHoodieDeltaStreamer > > > > Thoughts? > > > > Regards, > > Vinay Patil > > > --0000000000003073b305c4786bbf--