Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
From: =?utf-8?Q?J=C3=B6rn_Franke?= <jornfranke@gmail.com>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: Hive Context: Hive Metastore Client
Message-Id: <D4C69505-33AD-4F2D-866B-5FE211B6B1B0@gmail.com>
Date: Wed, 9 Mar 2016 17:14:03 +0100
References: 
 <CANM4JHAYj+zaza38MmvohDEaNGDVB=6Ox78fngw2EHhNhqVyuA@mail.gmail.com>
 <CAJ3fcbCNT7MpEAT9BWXXG8niwfjnTCfEqaemYE1fsEaK+29aFQ@mail.gmail.com>
 <56DF6014.9090604@gmail.com>
 <CAJ3fcbDkjW60dm-+Hxe2MFzPqBG8cvOQ89-U3V+_QHYoGcDWxg@mail.gmail.com>
 <D29A3AB0-C5E8-413B-B4A9-F78CE3E3A271@gmail.com>
In-Reply-To: <D29A3AB0-C5E8-413B-B4A9-F78CE3E3A271@gmail.com>
To: user@hive.apache.org

Apache Knox for authentication makes sense. For Hive authorization there are=
 tools such as Apache ranger or Sentry, which themselves can connect via LDA=
P.

> On 09 Mar 2016, at 16:58, Alan Gates <alanfgates@gmail.com> wrote:
>=20
> One way people have gotten around the lack of LDAP connectivity in HS2 has=
 been to use Apache Knox.  That project=E2=80=99s goal is to provide a singl=
e login capability for Hadoop related projects so that users can tie their L=
DAP or Active Directory servers into Hadoop.
>=20
> Alan.
>=20
>> On Mar 8, 2016, at 16:00, Mich Talebzadeh <mich.talebzadeh@gmail.com> wro=
te:
>>=20
>> The current scenario resembles a three tier architecture but without the s=
ecurity of second tier. In a typical three-tier you have users connecting to=
 the application server (read Hive server2) are independently authenticated a=
nd if OK, the second tier creates new ,NET type or JDBC threads to connect t=
o database much like multi-threading. The problem I believe is that Hive ser=
ver 2 does not have that concept of handling the individual loggings yet. Hi=
ve server 2 should be able to handle LDAP logins as well. It is a useful lay=
er to have.
>>=20
>> Dr Mich Talebzadeh
>>=20
>> LinkedIn  https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrbJd=
6zP6AcPCCdOABUrV8Pw
>>=20
>> http://talebzadehmich.wordpress.com
>>=20
>>=20
>> On 8 March 2016 at 23:28, Alex <this.side.of.confusion@gmail.com> wrote:
>> Yes, when creating a Hive Context a Hive Metastore client should be creat=
ed with a user that the Spark application will talk to the *remote* Hive Met=
astore with. We would like to add a custom authorization plugin to our remot=
e Hive Metastore to authorize the query requests that the spark application i=
s submitting which would also add authorization for any other applications h=
itting the Hive Metastore. Furthermore we would like to extend this so that w=
e can submit "jobs" to our Spark application that will allow us to run again=
st the metastore as different users while leveraging the abilities of our sp=
ark cluster. But as you mentioned only one login connects to the Hive Metast=
ore is shared among all HiveContext sessions.
>>=20
>> Likely the authentication would have to be completed either through a sec=
ured Hive Metastore (Kerberos) or by having the requests go through HiveServ=
er2.
>>=20
>> --Alex
>>=20
>>=20
>>> On 3/8/2016 3:13 PM, Mich Talebzadeh wrote:
>>> Hi,
>>>=20
>>> What do you mean by Hive Metastore Client? Are you referring to Hive ser=
ver login much like beeline?
>>>=20
>>> Spark uses hive-site.xml to get the details of Hive metastore and the lo=
gin to the metastore which could be any database. Mine is Oracle and as far a=
s I know even in  Hive 2, hive-site.xml has an entry for javax.jdo.option.Co=
nnectionUserName that specifies username to use against metastore database. T=
hese are all multi-threaded JDBC connections to the database, the same login=
 as shown below:
>>>=20
>>> LOGIN    SID/serial# LOGGED IN S HOST       OS PID         Client PID   =
  PROGRAM               MEM/KB      Logical I/O Physical I/O ACT
>>> -------- ----------- ----------- ---------- -------------- -------------=
- --------------- ------------ ---------------- ------------ ---
>>> INFO
>>> -------
>>> HIVEUSER 67,6160     08/03 08:11 rhes564    oracle/20539   hduser/1234  =
  JDBC Thin Clien        1,017               37            0 N
>>> HIVEUSER 89,6421     08/03 08:11 rhes564    oracle/20541   hduser/1234  =
  JDBC Thin Clien        1,081              528            0 N
>>> HIVEUSER 112,561     08/03 10:45 rhes564    oracle/24624   hduser/1234  =
  JDBC Thin Clien          889               37            0 N
>>> HIVEUSER 131,8811    08/03 08:11 rhes564    oracle/20543   hduser/1234  =
  JDBC Thin Clien        1,017               37            0 N
>>> HIVEUSER 47,30114    08/03 10:45 rhes564    oracle/24626   hduser/1234  =
  JDBC Thin Clien        1,017               37            0 N
>>> HIVEUSER 170,8955    08/03 08:11 rhes564    oracle/20545   hduser/1234  =
  JDBC Thin Clien        1,017              323            0 N
>>>=20
>>> As I understand what you are suggesting is that each Spark user uses dif=
ferent login to connect to Hive metastore. As of now there is only one login=
 that connects to Hive metastore shared among all
>>>=20
>>> 2016-03-08T23:08:01,890 INFO  [pool-5-thread-72]: HiveMetaStore.audit (H=
iveMetaStore.java:logAuditEvent(280)) - ugi=3Dhduser      ip=3D50.140.197.21=
7       cmd=3Dsource:50.140.197.217 get_table : db=3Dtest tbl=3Dt
>>> 2016-03-08T23:18:10,432 INFO  [pool-5-thread-81]: HiveMetaStore.audit (H=
iveMetaStore.java:logAuditEvent(280)) - ugi=3Dhduser      ip=3D50.140.197.21=
6       cmd=3Dsource:50.140.197.216 get_tables: db=3Dasehadoop pat=3D.*
>>>=20
>>> And this is an entry in Hive log when connection is made theough Zeppeli=
n UI
>>>=20
>>> 2016-03-08T23:20:13,546 INFO  [pool-5-thread-84]: metastore.HiveMetaStor=
e (HiveMetaStore.java:newRawStore(499)) - 84: Opening raw store with impleme=
ntation class:org.apache.hadoop.hive.metastore.ObjectStore
>>> 2016-03-08T23:20:13,547 INFO  [pool-5-thread-84]: metastore.ObjectStore (=
ObjectStore.java:initialize(318)) - ObjectStore, initialize called
>>> 2016-03-08T23:20:13,550 INFO  [pool-5-thread-84]: metastore.MetaStoreDir=
ectSql (MetaStoreDirectSql.java:<init>(142)) - Using direct SQL, underlying D=
B is ORACLE
>>> 2016-03-08T23:20:13,550 INFO  [pool-5-thread-84]: metastore.ObjectStore (=
ObjectStore.java:setConf(301)) - Initialized ObjectStore
>>>=20
>>> I am not sure there is currently such plan to have different logins allo=
wed to Hive Metastore. But it will add another level of security. Though I a=
m not sure how this would be authenticated.
>>>=20
>>> HTH
>>>=20
>>>=20
>>>=20
>>> Dr Mich Talebzadeh
>>>=20
>>> LinkedIn  https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrbJ=
d6zP6AcPCCdOABUrV8Pw
>>>=20
>>> http://talebzadehmich.wordpress.com
>>>=20
>>>=20
>>> On 8 March 2016 at 22:23, Alex F <this.side.of.confusion@gmail.com> wrot=
e:
>>> As of Spark 1.6.0 it is now possible to create new Hive Context sessions=
 sharing various components but right now the Hive Metastore Client is share=
d amongst each new Hive Context Session.
>>>=20
>>> Are there any plans to create individual Metastore Clients for each Hive=
 Context?
>>>=20
>>> Related to the question above are there any plans to create an interface=
 for customizing the username that the Metastore Client uses to connect to t=
he Hive Metastore? Right now it either uses the user specified in an environ=
ment variable or the application's process owner.
>=20