From dev-return-703-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Mon Jun 24 21:57:45 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 938C1180671 for ; Mon, 24 Jun 2019 23:57:45 +0200 (CEST) Received: (qmail 38220 invoked by uid 500); 24 Jun 2019 21:57:45 -0000 Mailing-List: contact dev-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list dev@hudi.apache.org Received: (qmail 38208 invoked by uid 99); 24 Jun 2019 21:57:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jun 2019 21:57:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 38467C2F02 for ; Mon, 24 Jun 2019 21:57:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.972 X-Spam-Level: ** X-Spam-Status: No, score=2.972 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.972] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ygaBO7ebOqHp for ; Mon, 24 Jun 2019 21:57:42 +0000 (UTC) Received: from sonic308-2.consmr.mail.bf2.yahoo.com (sonic308-2.consmr.mail.bf2.yahoo.com [74.6.130.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 04AEE5FE5F for ; Mon, 24 Jun 2019 21:57:41 +0000 (UTC) X-YMail-OSG: bqA1mNEVM1mtat8Wng9i4xA3gYbstOkkf4VBQ0b52SKBaqFXYJB1cYsYc6W6rnQ gBHG5OOuD060IctdxLh.tjlQ_quSW.uzmHpIJE82qjvoL9FquVakNkUMNNuOlCGCmqfAoc9Ikd9w 6nRHAI2MOfRI5.pcCK7ooOyvtKzWrQL3J3ohbp4nTPI91.1_gYfJn_tMBpHpQ5h5mxHmaULDSNaO .IjBJzxAlArVkCSccnzJ0idadevCRGRZ0gyXHVITUD8lnkWp_henIkIFCawf2WMLxF8vH.q0PdCo jqE05YNpLsdL2sSYb27iUhqW5VAHidlz3PZ4wekBOTPPZ1FdFfYWVvMnbHu3D9fayalsnr8TZaaT wwpd0C78IlUurBJebxaOeH_NIQ_ZGRn02gNQuODmEk7DJ14k_0pMmpDpBhFKq2nMr9.TiWV2Ganj .MuXwHO5v4Wo28oYMTwPmU0OLfEpjbn7eZmZ6hxztiz6PxdBx.rAa6sDOIK0.1USqns22DEIp8.7 dPtvxA4cmJsJas.fZnv.J9yEhyk8JsKdqCySXLo3m43uCMHcceCOnwTf8Ajl_IaE_eHUydXOdKOz u21FYM8.iIRtHHCGo0VEm2LEjQOYfyhs8pg7JX9ahIk2Rip.Ymj7E39.FIj.64.7Tl38G8UsYo4e i3juiy_Mw4dq1HmQoav5.i5.D9E98PU2gXcg3owC3YSgz4ZfRdaG1EpTgO6fSfTAu5TAhmhTfQ5o DauMCsO0HVgtqq5xRKYt51zsMLLNFkgnXGQJ.J28ElvS_c64HoRvwb2NAvlB01xlEUjvyBKUpVpA 7lNOGRs8DtsSFfYKIplxfcmvXZv5r7tHeDjc8z1wDeBVcE5RLRYLzHNguv_gnlQZK3cEIF5M7UxQ US4lzOKM7jHlexDdKFEB7RHPr5rFlMOnJo9lgT.Ch.6OlBWer_vGpqeaxoaox1EqwvUjSjd90mYv 0LhtVpiPO.aoDE.p2K0BqDQXu96Z_ax15dddzue_NGs_xPjWISFA1ERIN.RrlKtvzb0D6S0upJIq 6.NPKQTROwpq651l6PWmn42E_OgYYaePSc4Z30dW1iRPvDErE.U.0rLPdkJOQN5dTEppnro83q5u 4EKdzxI2KEQwDhT9fheG0nI8pxOx6bgmPS5JfWObseO3SOeJjkczlzXJID1Bf3IdLHFJvSxsAjhc etj5KPeuwsDS.PFcc_V7zUv83slfiPCvSZOdX1DH2jvCHlMroFtB741BnaeObbRNKBAt2yIRl5gi X Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.bf2.yahoo.com with HTTP; Mon, 24 Jun 2019 21:57:41 +0000 Date: Mon, 24 Jun 2019 21:57:36 +0000 (UTC) From: "vbalaji@apache.org" To: dev@hudi.apache.org Message-ID: <1252751165.834264.1561413456543@mail.yahoo.com> In-Reply-To: <2091735623.835082.1561409735544@mail.yahoo.com> References: <2091735623.835082.1561409735544@mail.yahoo.com> Subject: Re: Hoodie dataset write without partition MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_834263_243379578.1561413456541" X-Mailer: WebService/1.1.13837 YMailNorrin Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36 ------=_Part_834263_243379578.1561413456541 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable =20 Hi Amarnath, Apart from changing the partition extractor class, you would need to change= the keyGeneratorClass for non-partitioned table. Use this param "--key-generator-class com.uber.hoodie.NonpartitionedKeyGene= rator"=C2=A0as part of DeltaStreamer command-line execution. Also, ensure we have the following configs defined in the properties file u= sed by delta-streamer: hoodie.datasource.write.keygenerator.class=3Dcom.uber.hoodie.Nonpartitioned= KeyGeneratorhoodie.datasource.hive_sync.partition_extractor_class=3Dcom.ube= r.hoodie.hive.NonPartitionedExtractorWe will eventually remove the DeltaStr= eamer CLI and rely on the properties config for uniform handling. Thanks,Balaji.V On Monday, June 24, 2019, 1:55:51 PM PDT, Balaji Varadarajan wrote: =20 =20 Hi Amarnath, I will look into it and reply back by EOD today. Balaji.V =C2=A0 =C2=A0 On Sunday, June 23, 2019, 8:21:51 AM PDT, Amarnath Venkataswa= my wrote:=C2=A0=20 =20 Hi=20 Is there any option to write the hoodie dataset without any partition? I tried but hive sync is failing when you sync up without any partition. Delta streamer creates with default as partition when there is no partition= column. Sent from my iPhone=C2=A0 =20 ------=_Part_834263_243379578.1561413456541--