ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Don Bosco Durai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
Date Sat, 14 Jul 2018 08:37:00 GMT

    [ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544118#comment-16544118
] 

Don Bosco Durai commented on RANGER-2128:
-----------------------------------------

[~Qin Yao] I managed to get your code working with Ranger master branch. I have attached the
updates I have done. Can you try it on your side and see whether it works for you?

It works with Spark Shell, but with Thrift Server I am getting an exception. Have you seen
this before? I have enabled Ranger in thrift server by adding the following line in sparkconf.conf
{code:java}
spark.sql.extensions org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension
{code}


Error opening session: 
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: org.apache.hadoop.hive.metastore.ObjectStore cannot be cast
to org.apache.hadoop.hive.metastore.RawStore;
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
	at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:138)
	at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:133)
	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:54)
	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:54)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:252)
	at org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
	at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3253)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)
	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
	at org.apache.hive.service.cli.CLIService.openSession(CLIService.java:194)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:354)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:246)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

> Implement SparkSQL plugin
> -------------------------
>
>                 Key: RANGER-2128
>                 URL: https://issues.apache.org/jira/browse/RANGER-2128
>             Project: Ranger
>          Issue Type: New Feature
>          Components: plugins, Ranger
>    Affects Versions: 1.1.0
>            Reporter: t oo
>            Assignee: Kent Yao
>            Priority: Major
>             Fix For: 2.0.0
>
>
> Implement SparkSQL plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message