Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5AEFD17AF1 for ; Tue, 14 Oct 2014 22:00:20 +0000 (UTC) Received: (qmail 19300 invoked by uid 500); 14 Oct 2014 22:00:18 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 19235 invoked by uid 500); 14 Oct 2014 22:00:18 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 19224 invoked by uid 99); 14 Oct 2014 22:00:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Oct 2014 22:00:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of leftyleverenz@gmail.com designates 209.85.223.180 as permitted sender) Received: from [209.85.223.180] (HELO mail-ie0-f180.google.com) (209.85.223.180) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Oct 2014 22:00:13 +0000 Received: by mail-ie0-f180.google.com with SMTP id x19so13344ier.11 for ; Tue, 14 Oct 2014 14:59:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=MMAFDtp3Ti29+/H432xVym/cVQUC7gGZ7GUFuGtMwSo=; b=hX/4wwNuBcVKgkRs2zbHKmDC6XfzWDja6bswrFszf4ftaNFsIeV2wnkIPWNpUCc/SS P06dUPh42CQGtOJ2eiLmSyubqmBW3IDigbfLfz0T/BmYNfgtvFqOn81HoZcqdm5inPN4 tTCYZur+hHxpu57mdWCaCHZL/FEZeDQFV6CtF17tIOPjBXfXIoB12U6jlu8xH4AElHTA sXR3GrwzRZzzk6C0Jrnf/1fKnlN4V6Dovcd7ZmXinhJ/PMHIcZAYkiZscRJPxBbIAsyZ Vx5Bt7eghZt70JLSz4qaFs8OZEhW8CeRLj/N4HiIQ/uKwLiUtyBMUFJMOBfA9gmhopsb QGUQ== MIME-Version: 1.0 X-Received: by 10.50.112.70 with SMTP id io6mr2986241igb.15.1413323993357; Tue, 14 Oct 2014 14:59:53 -0700 (PDT) Received: by 10.42.91.134 with HTTP; Tue, 14 Oct 2014 14:59:53 -0700 (PDT) In-Reply-To: References: Date: Tue, 14 Oct 2014 17:59:53 -0400 Message-ID: Subject: Re: Java api for connecting to hiveserver2 From: Lefty Leverenz To: user@hive.apache.org Content-Type: multipart/alternative; boundary=047d7b3a97ea93a35a0505692273 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b3a97ea93a35a0505692273 Content-Type: text/plain; charset=UTF-8 That's an older version of the HCatalog javadocs. You can find current javadocs here: http://hive.apache.org/javadocs/r0.13.1/api/index.html. -- Lefty On Sun, Oct 12, 2014 at 3:05 PM, Suhas Gogate wrote: > Edward/Hanish, this is a reasonable use case to justify Java API. I > initially thought need for Java API to avoid knowing SQL :) but that you > can't for any DML operations :) So I guess for DDL operations you should > transition to HCatalog API.. See the design document where first para > explicitly advises it.. > > > https://cwiki.apache.org/confluence/display/HCATALOG/Design+Document+-+Java+APIs+for+HCatalog+DDL+Commands > > I have not played as much with HCatalog APIs yet and quick look tells me > that it is not at par in terms of all DDL/auth operations yet! > > https://hive.apache.org/javadocs/hcat-r0.5.0/api/index.html > (org.apache.hcatalog.api.HCatClient). > > --Suhas > > > On Sat, Oct 11, 2014 at 2:52 PM, Hanish Bansal < > hanish.bansal.agarwal@gmail.com> wrote: > >> Thanks all for response !! >> >> Totally agreed with Edward. >> >> Also the reason why we don't want to use jdbc client is: Writing sql >> statement for everything is little bit complex rather than calling direct >> java apis. >> With java metastore client api it was quite easy. >> >> Regards, >> Hanish >> On 12/10/2014 2:25 am, "Edward Capriolo" wrote: >> >>> Actually I do not love the jdbc api for all things. >>> >>> I used to have several scripts that looped over all database, all >>> tables, and looked at specific meta store properties or locations. These >>> scripts were very small and to the point sometimes 5 line loops. Now I have >>> to write things like this: >>> >>> ResultSet r = Statement.executeQuery("describe exteneded "+table) >>> while (r.next){ >>> if (r.get(1).getString().equals("location")... >>> ... >>> >>> It is more clunky then using the metastore api directly: >>> >>> hiveClient.getTable("database", "table").getSD().getLocation(); >>> >>> >>> >>> On Sat, Oct 11, 2014 at 4:29 PM, Vaibhav Gumashta < >>> vgumashta@hortonworks.com> wrote: >>> >>>> Hanish, >>>> >>>> I agree with Suhas and would strongly encourage you to use the JDBC API >>>> for HiveServer2. HiveServer2 has a thrift api for the client-server RPC, >>>> but that is *not* intended for end user consumption and could end up >>>> breaking your current code in future. >>>> >>>> Is there any specific feature you are looking at which Hive's JDBC >>>> driver doesn't implement? >>>> >>>> Thanks, >>>> --Vaibhav >>>> >>>> On Sat, Oct 11, 2014 at 12:43 PM, Suhas Gogate >>>> wrote: >>>> >>>>> Sorry Anish, but being database programmer in the past, I always used >>>>> embedded SQL interface... I was wondering should we really need direct Java >>>>> interface w/ HiveServer2? May be I am wrong, but would like to know your >>>>> view on what are the limitations of using embedded SQL vs direct Java API. >>>>> >>>>> --Suhas >>>>> >>>>> On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate >>>>> wrote: >>>>> >>>>>> Hanish, this is interesting question and I also faced similar >>>>>> limitation lately. Although as Hive getting more closer to relational model >>>>>> with richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way >>>>>> to invoke embedded SQL in Java, the real question is should Hive Metastore >>>>>> Client (java) API should at all be used by user, rather all the existing >>>>>> Hive Client interfaces be talking to Hive Metastore internally? >>>>>> >>>>>> --Suhas >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal < >>>>>> hanish.bansal.agarwal@gmail.com> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I am planning to use sql based authorization that is recently >>>>>>> introduced in hive 0.13.0. >>>>>>> >>>>>>> I was using hive metastore client java api early which has direct >>>>>>> apis for all operations like grant, revoke etc. >>>>>>> But for using new authorization I ll have to use hiveserver2 and >>>>>>> pass all requests through hiveserver2. I came up with JDBC client which can >>>>>>> be used to connect hiveserver2. The restriction there is we must have to >>>>>>> write SQL statements. >>>>>>> >>>>>>> I want to know is there any java api to connect hiveserver2, that >>>>>>> have direct java methods to perform operations and we don't need to write >>>>>>> SQL statements ?? >>>>>>> >>>>>>> Thanks, >>>>>>> Hanish >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to which it is addressed and may contain information that is >>>> confidential, privileged and exempt from disclosure under applicable law. >>>> If the reader of this message is not the intended recipient, you are hereby >>>> notified that any printing, copying, dissemination, distribution, >>>> disclosure or forwarding of this communication is strictly prohibited. If >>>> you have received this communication in error, please contact the sender >>>> immediately and delete it from your system. Thank You. >>> >>> >>> > --047d7b3a97ea93a35a0505692273 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
That's an older version of the HCatalog javadocs.=C2= =A0 You can find current javadocs here: =C2=A0http://hive.apache.org/javadocs/r0.13= .1/api/index.html.

-- Lefty

On Sun, Oct 12, 2014 at 3:05 PM, Suhas Gogat= e <vgogate@pivotal.io> wrote:
Edward/Hanish, this is a reasonable use case to justi= fy Java API. I initially thought need for Java API to avoid knowing SQL :) = but that you can't for any DML operations :) =C2=A0So I guess for DDL o= perations you should transition to HCatalog API.. See the design document w= here first para explicitly advises it..


I have not played as much with HCatalog= APIs yet and quick look tells me that it is not at par in terms of all DDL= /auth operations yet!

https://hive.= apache.org/javadocs/hcat-r0.5.0/api/index.html (org.apache.hcatalog.api= .HCatClient).=C2=A0

--Suhas


On Sat, Oct 11, 2014 at 2:52 PM, Hanish Bansal <hanish.bansal.agarwal@gmail.com> wrote:

Thanks all for response !!

Totally agreed with Edward.

Also the reason why we don't want to use jdbc client is:= Writing sql statement for everything is little bit complex rather than cal= ling direct java apis.
With java metastore client api it was quite easy.

Regards,
Hanish

On 12/10/2014 2:25 am, "Edward Capriolo&quo= t; <edlinuxgu= ru@gmail.com> wrote:
Actually I do not lo= ve the jdbc api for all things.

I used to have several scripts= that looped over all database, all tables, and looked at specific meta sto= re properties or locations. These scripts were very small and to the point = sometimes 5 line loops. Now I have to write things like this:

= ResultSet r =3D Statement.executeQuery("describe exteneded "+tabl= e)
while (r.next){
if (r.get(1).getString().equals("= location")...
...

It is more clunky then using the met= astore api directly:

hiveClient.getTable("database",= "table").getSD().getLocation();



On Sat, Oct 11, 2014 at 4:29 PM,= Vaibhav Gumashta <vgumashta@hortonworks.com> wrote:=
Hanish,

<= div>I agree with Suhas and would strongly encourage you to use the JDBC API= for HiveServer2. HiveServer2 has a thrift api for the client-server RPC, b= ut that is not intended for end user consumption and could end up br= eaking your current code in future.

Is there any s= pecific feature you are looking at which Hive's JDBC driver doesn't= implement?

Thanks,
--Vaibhav

On Sat, Oct 11, = 2014 at 12:43 PM, Suhas Gogate <vgogate@pivotal.io> wrote:<= br>
Sorry Anish, but being d= atabase programmer in the past, I always used embedded SQL interface... I w= as wondering should we really need direct Java interface w/ HiveServer2? Ma= y be I am wrong, but would like to know your view on what are the limitatio= ns of using embedded SQL vs direct Java API.=C2=A0

--Suhas =C2=A0
=

On Sat, Oct 11, 2= 014 at 12:38 PM, Suhas Gogate <vgogate@pivotal.io> wrote:
Hanish, this is interesti= ng question and I also faced similar limitation lately. Although as Hive ge= tting more closer to relational model with richer SQL interface (DDL/Author= ization, DML) and HiveServer2 as a way to invoke embedded SQL in Java, the = real question is should Hive Metastore Client (java) API should at all be u= sed by user, rather all the existing Hive Client interfaces be talking to H= ive Metastore internally?=C2=A0

--Suhas



On Fri, O= ct 10, 2014 at 9:21 PM, Hanish Bansal <hanish.bansal.agarwal= @gmail.com> wrote:

Hi All,

I am planning to use sql based authorization that is recentl= y introduced in hive 0.13.0.

I was using hive metastore client java api early which has d= irect apis for all operations like grant, revoke etc.
But for using new authorization I ll have to use hiveserver2 and pass all r= equests through hiveserver2. I came up with JDBC client which can be used t= o connect hiveserver2. The restriction there is we must have to write SQL s= tatements.

I want to know is there any java api to connect hiveserver2,= that have direct java methods to perform operations and we don't need = to write SQL statements ??

Thanks,
Hanish





CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.



--047d7b3a97ea93a35a0505692273--