hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot West <tea...@gmail.com>
Subject Client agnostic metastore authorization in AWS
Date Mon, 24 Apr 2017 09:59:11 GMT
We’re operating long lived Hive metastore instances in AWS to provide a
metadata source of truth for our data processing pipelines. These pipelines
are not restricted to Hive SQL, but must use frameworks that can integrate
with the metastore (such as Spark). We’re storing data in S3. As these are
centralized shared resources, access control is a concern. Primarily this
to guard against user error rather than malicious intent (e.g. user
accidentally drops the wrong table). We’ve been experimenting with Kerberos
in EMR for strong authentication but have no definitive solution for
authorization as yet. We cannot use SQL based authorization as I believe
this is implemented at HiveServer2, which is not a useful integration point
for us. Additionally, there is no workable implementation for metastore
storage based authorization on S3. We’ve toyed with a
HiveMetastoreAuthorizationProvider that evaluates S3 bucket policies, but
this is fairly complex, especially when mapping principles to IAM entities.

More recently we came across DefaultHiveMetastoreAuthorizationProvider in
our travels which appears to implement legacy GRANT/REVOKE type controls in
both the Hive client and metastore layers. I’ve managed to get this working
solely in the metastore, and control access to metadata entities both via
the Hive CLI and the Thrift API. For example, I can craft some Thrift calls
to drop a table that correctly fail with one UGI principal, but succeed
with another. My metastore service configuration is like so, note that no
client side authorization plugin is configured:

hive.security.authorization.enabled=true
hive.security.authorization.createtable.owner.grants=ALL
hive.metastore.pre.event.listeners=org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
hive.security.metastore.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider


Aside from the documented shortcomings of this legacy authorization
implementation, can anyone suggest any additional pitfalls with this
configuration?

Thanks,

Elliot.

Mime
View raw message