spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-13983) HiveThriftServer2 can not get "--hiveconf" or ''--hivevar" variables since 1.6 version (both multi-session and single session)
Date Thu, 01 Feb 2018 18:38:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiao Li resolved SPARK-13983.
-----------------------------
       Resolution: Fixed
         Assignee: Yuming Wang  (was: Cheng Lian)
    Fix Version/s: 2.3.0

> HiveThriftServer2 can not get "--hiveconf" or ''--hivevar" variables since 1.6 version
(both multi-session and single session)
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-13983
>                 URL: https://issues.apache.org/jira/browse/SPARK-13983
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0, 1.6.1
>         Environment: ubuntu, spark 1.6.0 standalone, spark 1.6.1 standalone
> (tried spark branch-1.6 snapshot as well)
> compiled with scala 2.10.5 and hadoop 2.6
> (-Phadoop-2.6 -Psparkr -Phive -Phive-thriftserver)
>            Reporter: Teng Qiu
>            Assignee: Yuming Wang
>            Priority: Major
>             Fix For: 2.3.0
>
>
> HiveThriftServer2 should be able to get "\--hiveconf" or ''\-\-hivevar" variables from
JDBC client, either from command line parameter of beeline, such as
> {{beeline --hiveconf spark.sql.shuffle.partitions=3 --hivevar db_name=default}}
> or from JDBC connection string, like
> {{jdbc:hive2://localhost:10000?spark.sql.shuffle.partitions=3#db_name=default}}
> this worked in spark version 1.5.x, but after upgraded to 1.6, it doesn't work.
> to reproduce this issue, try to connect to HiveThriftServer2 with beeline:
> {code}
> bin/beeline -u jdbc:hive2://localhost:10000 \
>             --hiveconf spark.sql.shuffle.partitions=3 \
>             --hivevar db_name=default
> {code}
> or
> {code}
> bin/beeline -u jdbc:hive2://localhost:10000?spark.sql.shuffle.partitions=3#db_name=default
> {code}
> will get following results:
> {code}
> 0: jdbc:hive2://localhost:10000> set spark.sql.shuffle.partitions;
> +-------------------------------+--------+--+
> |              key              | value  |
> +-------------------------------+--------+--+
> | spark.sql.shuffle.partitions  | 200    |
> +-------------------------------+--------+--+
> 1 row selected (0.192 seconds)
> 0: jdbc:hive2://localhost:10000> use ${db_name};
> Error: org.apache.spark.sql.AnalysisException: cannot recognize input near '$' '{' 'db_name'
in switch database statement; line 1 pos 4 (state=,code=0)
> {code}
> -
> but this bug does not affect current versions of spark-sql CLI, following commands works:
> {code}
> bin/spark-sql --master local[2] \
>               --hiveconf spark.sql.shuffle.partitions=3 \
>               --hivevar db_name=default
> spark-sql> set spark.sql.shuffle.partitions
> spark.sql.shuffle.partitions   3
> Time taken: 1.037 seconds, Fetched 1 row(s)
> spark-sql> use ${db_name};
> OK
> Time taken: 1.697 seconds
> {code}
> so I think it may caused by this change: https://github.com/apache/spark/pull/8909 (
[SPARK-10810] [SPARK-10902] [SQL] Improve session management in SQL )
> perhaps by calling {{hiveContext.newSession}}, the variables from {{sessionConf}} were
not loaded into the new session? (https://github.com/apache/spark/pull/8909/files#diff-8f8b7f4172e8a07ff20a4dbbbcc57b1dR69)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message