ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kent Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
Date Wed, 27 Jun 2018 11:30:00 GMT

    [ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524935#comment-16524935
] 

Kent Yao commented on RANGER-2128:
----------------------------------

[~bosco] spark-authorizer works for spark2.1 and later, this pull request is compatible with
 *2.2 and later * for supporting spark.sql.extensions configuration.

I had it tested with my ranger 0.5-based personal release version against 2.2.1, and works
fine.

START 
{code:java}
bin/spark-shell --proxy-user hzyaoqin --conf spark.sql.warehouse.dir=/user/hzyaoqin/warehouse
--conf spark.sql.extensions=org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension
{code}

TEST
{code:java}
scala> sc.setLogLevel("info");

scala> sql("show tables").show
{code}

LOG Snippet

{code:java}
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-plugins-cred-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/jersey-client-1.14.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/eclipselink-2.5.2.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/gson-2.2.4.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-plugins-common-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-hive-plugin-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/httpcore-4.4.6.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-plugins-audit-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/solr-solrj-5.5.5.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/noggit-0.6.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/httpclient-4.5.3.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/mysql-connector-java-5.1.39.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/httpmime-4.5.3.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger_solrj-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/javax.persistence-2.1.0.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil: getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding /home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/mybatis-3.2.8.jar
{code}


PS, I'd like to know the latest version which supports hive1.2.1 by ranger-hive-plugin  to
build spark-plugin assembly 
Thanks



> Implement SparkSQL plugin
> -------------------------
>
>                 Key: RANGER-2128
>                 URL: https://issues.apache.org/jira/browse/RANGER-2128
>             Project: Ranger
>          Issue Type: New Feature
>          Components: plugins, Ranger
>    Affects Versions: 1.1.0
>            Reporter: t oo
>            Assignee: Kent Yao
>            Priority: Major
>             Fix For: 1.1.0
>
>
> Implement SparkSQL plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message