spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lokesh Yadav (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-18832) Spark SQL: Thriftserver unable to run a registered Hive UDTF
Date Thu, 23 Feb 2017 08:32:44 GMT

     [ https://issues.apache.org/jira/browse/SPARK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lokesh Yadav updated SPARK-18832:
---------------------------------
    Description: 
Spark Thriftserver is unable to run a HiveUDTF.
It throws the error that it is unable to find the functions although the function registration
succeeds and the funtions does show up in the list output by {{show functions}}.

I am using a Hive UDTF, registering it using a jar placed on my local machine. Calling it
using the following commands:

//Registering the functions, this command succeeds.
{{CREATE FUNCTION SampleUDTF AS 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING
JAR '/root/spark_files/experiments-1.2.jar';}}

//Thriftserver is able to look up the functuion, on this command:
{{DESCRIBE FUNCTION SampleUDTF;}}

{quote}
{noformat}
Output: 
+-----------------------------------------------------------+--+
|                       function_desc                       |
+-----------------------------------------------------------+--+
| Function: default.SampleUDTF                              |
| Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF  |
| Usage: N/A.                                               |
+-----------------------------------------------------------+--+
{noformat}
{quote}

// Calling the function: 
{{SELECT SampleUDTF('Paris');}}

bq. Output of the above command: Error: org.apache.spark.sql.AnalysisException: Undefined
function: 'SampleUDTF'. This function is neither a registered temporary function nor a permanent
function registered in the database 'default'.; line 1 pos 7 (state=,code=0)


I have also tried with using a non-local (on hdfs) jar, but I get the same error.

My environment: HDP 2.4 with spark 2.00




  was:


Spark Thriftserver is unable to run a HiveUDTF.
It throws the error that it is unable to find the functions although the function registration
succeeds and the funtions does show up in the list output by {{show functions}}.

I am using a Hive UDTF, registering it using a jar placed on my local machine. Calling it
using the following commands:

//Registering the functions, this command succeeds.
{{CREATE FUNCTION SampleUDTF AS 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING
JAR '/root/spark_files/experiments-1.2.jar';}}

//Thriftserver is able to look up the functuion, on this command:
{{DESCRIBE FUNCTION SampleUDTF;}}

{quote}
Output: 
+-----------------------------------------------------------+--+
|                       function_desc                       |
+-----------------------------------------------------------+--+
| Function: default.SampleUDTF                              |
| Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF  |
| Usage: N/A.                                               |
+-----------------------------------------------------------+--+
{quote}

// Calling the function: 
{{SELECT SampleUDTF('Paris');}}

bq. Output of the above command: Error: org.apache.spark.sql.AnalysisException: Undefined
function: 'SampleUDTF'. This function is neither a registered temporary function nor a permanent
function registered in the database 'default'.; line 1 pos 7 (state=,code=0)


I have also tried with using a non-local (on hdfs) jar, but I get the same error.

My environment: HDP 2.4 with spark 2.00





> Spark SQL: Thriftserver unable to run a registered Hive UDTF
> ------------------------------------------------------------
>
>                 Key: SPARK-18832
>                 URL: https://issues.apache.org/jira/browse/SPARK-18832
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0, 2.0.1, 2.0.2
>         Environment: HDP: 2.5
> Spark: 2.0.0
>            Reporter: Lokesh Yadav
>         Attachments: SampleUDTF.java
>
>
> Spark Thriftserver is unable to run a HiveUDTF.
> It throws the error that it is unable to find the functions although the function registration
succeeds and the funtions does show up in the list output by {{show functions}}.
> I am using a Hive UDTF, registering it using a jar placed on my local machine. Calling
it using the following commands:
> //Registering the functions, this command succeeds.
> {{CREATE FUNCTION SampleUDTF AS 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING
JAR '/root/spark_files/experiments-1.2.jar';}}
> //Thriftserver is able to look up the functuion, on this command:
> {{DESCRIBE FUNCTION SampleUDTF;}}
> {quote}
> {noformat}
> Output: 
> +-----------------------------------------------------------+--+
> |                       function_desc                       |
> +-----------------------------------------------------------+--+
> | Function: default.SampleUDTF                              |
> | Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF  |
> | Usage: N/A.                                               |
> +-----------------------------------------------------------+--+
> {noformat}
> {quote}
> // Calling the function: 
> {{SELECT SampleUDTF('Paris');}}
> bq. Output of the above command: Error: org.apache.spark.sql.AnalysisException: Undefined
function: 'SampleUDTF'. This function is neither a registered temporary function nor a permanent
function registered in the database 'default'.; line 1 pos 7 (state=,code=0)
> I have also tried with using a non-local (on hdfs) jar, but I get the same error.
> My environment: HDP 2.4 with spark 2.00



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message