From user-return-21011-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Jul 3 12:20:38 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 335D918067B for ; Tue, 3 Jul 2018 12:16:47 +0200 (CEST) Received: (qmail 88714 invoked by uid 500); 3 Jul 2018 10:16:45 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 88704 invoked by uid 99); 3 Jul 2018 10:16:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jul 2018 10:16:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 82961C0024 for ; Tue, 3 Jul 2018 10:16:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.192 X-Spam-Level: X-Spam-Status: No, score=0.192 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1.697, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id BBN7e0hdh-Yx for ; Tue, 3 Jul 2018 10:16:43 +0000 (UTC) Received: from mail-oi0-f48.google.com (mail-oi0-f48.google.com [209.85.218.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 50C7B5F19D for ; Tue, 3 Jul 2018 10:16:43 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id n84-v6so2667309oib.9 for ; Tue, 03 Jul 2018 03:16:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yM2LN4Jz6tFPbL8KnYtOCJSb82NNRZH4Fl0drCnrFz8=; b=iy7MZT4fkgwoW8nRSt1ypu1MWzbc+j48EXfwzMCGcdot4NRnXrHqQHRXLHmkjBjhvq IyRfjO6hVKRWocsya1t0zHSUuFzaqA2H50mg3vSdJrXKBrHaX9cbSgf+04Ay/kiKa2Al INeI1EzJIcezEI8CgD19DgSBT+LH6j6dHfTlNge1IUdtlLwrLPPNzO3Jp9r4cLFiFdQ8 VLz+HCtV6zpZegHQh0i++IYDsaZj54ZYk0OpJp3Agw/iGQg8H1Qu6NE856L2wes5EyPR wSzVgUWQ4N+r0BDNe1dzHK38WNvBNp6Wy2fVgtGafWZ5scrVjwAuwi5Pt/iC9CoxclpC v+rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yM2LN4Jz6tFPbL8KnYtOCJSb82NNRZH4Fl0drCnrFz8=; b=CBSoAamOPzcXVWoprSYfdEoTui/cuEjc1yB4rfDwLy/L07vPFT1M4IA9+lgYcFYsmn BaKtK0CEnJNhbvqCl2wwCfJuNRNyCUIIXOeaMSoNw4swXfT57YVu/od4nT3FsqLFU0Kj 44YK9wae2r4ZSqVKOgeqX1wwDw7cuwQWlp1vETO6FBN8m3zWhSSetKxdn3cbktIvRLsi t3MES3BM5ELU6tV3gJPHLpY8EhC/iaPh0uc7UYT8RyWbExXAPxRtngr1r7QyaWKs1Vwg dAfcFsLyeMug5djU5G5RMSLUWJDo65LgBezB99OHEnSbYBFM1QFYko/fhTW00gAOYHF5 fCBQ== X-Gm-Message-State: APt69E1hghfTlSk2hMrdKWwA7df4e7HBBvDN+bgirk2aQiguy/Vm+OqM WMo8LX47h/B6LI0aIjJwbGIcBdhcgxIr3PAmHQBKrg== X-Google-Smtp-Source: AAOMgpdQzpsPYkjrPMmklRKOOCBoBrvQ7+wGTkJvwOl+lF5Ym3A0aPTLZkW7qQHiSjwHBt0ayv5yQG9I9mk0BdSRfg8= X-Received: by 2002:aca:686:: with SMTP id 128-v6mr20401092oig.31.1530613001969; Tue, 03 Jul 2018 03:16:41 -0700 (PDT) MIME-Version: 1.0 References: <88c43d5d-f512-cfc4-5d7d-e57be447bcaf@apache.org> <1b6c70dd-5f04-4452-8a68-6cb5d284b353@apache.org> In-Reply-To: <1b6c70dd-5f04-4452-8a68-6cb5d284b353@apache.org> From: Shivam Sharma <28shivamsharma@gmail.com> Date: Tue, 3 Jul 2018 15:46:30 +0530 Message-ID: Subject: Re: Regarding external metastore like HIVE To: twalthr@apache.org Cc: user@flink.apache.org Content-Type: multipart/alternative; boundary="0000000000004564ec0570159c34" --0000000000004564ec0570159c34 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Timo, Fabian I will follow this. Best On Tue, Jul 3, 2018 at 3:01 PM Timo Walther wrote: > Hi, > > you can follow the progress here: > https://issues.apache.org/jira/browse/FLINK-9171 > > Regards, > Timo > > > Am 03.07.18 um 10:32 schrieb Fabian Hueske: > > Hi, > > The docs explain that the ExternalCatalog interface *can* be used to > implement a catalog for HCatalog or Metastore. > However, there is no such implementation in Flink yet. You would need to > implement such as catalog connector yourself. > > I think there would be quite a few people interested in such a connector. > > Best, Fabian > > 2018-07-03 10:24 GMT+02:00 Shivam Sharma <28shivamsharma@gmail.com>: > >> Hi, >> >> Please find the documentation link here: >> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/co= mmon.html#register-an-external-catalog >> . >> >> Currently, Flink provides an InMemoryExternalCatalog for demo and >>> testing purposes. However, the ExternalCatalog interface can also be >>> used to connect catalogs like *HCatalog or Metastore* to the Table API. >> >> >> Thanks >> >> >> On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni wrote: >> >>> @ashwin >>> >>> I cannot find the documention about `metastore` >>> >>> Could you give the reference? >>> >>> Ashwin Sinha =E4=BA=8E2018=E5=B9=B47=E6=9C=88= 3=E6=97=A5=E5=91=A8=E4=BA=8C =E4=B8=8B=E5=8D=884:08=E5=86=99=E9=81=93=EF=BC= =9A >>> >>>> By metastore we mean catalog where table information is stored. >>>> >>>> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, wrote= : >>>> >>>>> What do you mean with "Metastore"? Are you referring to state backend= s >>>>> >>>>> ? >>>>> >>>>> On 02.07.2018 18:41, Shivam Sharma wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have read Flink documentation that Flink supports Metastore which i= s >>>>> currently InMemory. Is Flink community thinking to implement external >>>>> Metastore like Hive? >>>>> >>>>> Thanks >>>>> >>>>> -- >>>>> Shivam Sharma >>>>> Data Engineer @ Goibibo >>>>> Indian Institute Of Information Technology, Design and Manufacturing >>>>> Jabalpur >>>>> Mobile No- (+91) 8882114744 >>>>> Email:- 28shivamsharma@gmail.com >>>>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma >>>>> * >>>>> >>>>> >>>>> >>>> ::DISCLAIMER:: >>>> >>>> >>>> ----------------------------------------------------------------------= ---------------------------------------------------------------------------= --- >>>> >>>> >>>> This message is intended only for the use of the addressee and may >>>> contain information that is privileged, confidential and exempt from >>>> disclosure under applicable law. If the reader of this message is not = the >>>> intended recipient, or the employee or agent responsible for deliverin= g the >>>> message to the intended recipient, you are hereby notified that any >>>> dissemination, distribution or copying of this communication is strict= ly >>>> prohibited. If you have received this e-mail in error, please notify u= s >>>> immediately by return e-mail and delete this e-mail and all attachment= s >>>> from your system. >>>> >>> >> >> -- >> Shivam Sharma >> Data Engineer @ Goibibo >> Indian Institute Of Information Technology, Design and Manufacturing >> Jabalpur >> Mobile No- (+91) 8882114744 >> Email:- 28shivamsharma@gmail.com >> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma >> * >> > > > --=20 Shivam Sharma Data Engineer @ Goibibo Indian Institute Of Information Technology, Design and Manufacturing Jabalpur Mobile No- (+91) 8882114744 Email:- 28shivamsharma@gmail.com LinkedIn:-*https://www.linkedin.com/in/28shivamsharma * --0000000000004564ec0570159c34 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank= s Timo, Fabian
I will follow = this.

<= div class=3D"gmail_default" style=3D"color:#000000">Best

On Tue, Jul 3, 2018 at 3:01 PM Tim= o Walther <twalthr@apache.org&= gt; wrote:
=20 =20 =20
Hi,

you can follow the progress here: https://iss= ues.apache.org/jira/browse/FLINK-9171

Regards,
Timo


Am 03.07.18 um 10:32 schrieb Fabian Hueske:
Hi,

The docs explain that the ExternalCatalog interface can be used to implement a catalog for HCatalog or Metastore.
However, there is no such implementation in Flink yet. You would need to implement such as catalog connector yourself.

I think there would be quite a few people interested in such a connector.

Best, Fabian

2018-07-03 10:24 GMT+02:00 Shivam Sharma <28shivamsharma@gmail.com>:
Hi,

Pleas= e find the documentation link here:=C2=A0https://ci.apache.org/project= s/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-c= atalog.

Curre= ntly, Flink provides an=C2=A0InMem= oryExternalCatalog=C2=A0for demo and testing purposes. However, the=C2=A0ExternalCatalog=C2=A0interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.

Thanks


On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <nichuanlei@gmail.com> wrote:
@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <ashwin.sinha@go-mmt.com> =E4=BA=8E2018=E5=B9=B47=E6=9C=883=E6=97=A5=E5=91=A8= =E4=BA=8C =E4=B8=8B=E5=8D=884:08=E5=86=99=E9=81=93=EF=BC=9A
By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <chesnay@apache.org> wrote:
What do you mean with "Metastore"? Are= you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:
Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive?=C2=A0

Thanks

--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing Jabalpur
Mobile No-=C2=A0(+91) 8882114744



::DISCLAIMER:= :

-------------= ---------------------------------------------------------------------------= ------------------------------------------------------------


This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.



--
Shiva= m Sharma
Data Engineer @ Goibibo
India= n Institute Of Information Technology, Design and Manufacturing Jabalpur
Mobil= e No-=C2=A0(+91) 8882114744




--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing= Jabalpur
--0000000000004564ec0570159c34--