spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From marmbrus <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-6908] [SQL] Use isolated Hive client
Date Thu, 07 May 2015 21:51:36 GMT
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5876#discussion_r29899653
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
    @@ -93,9 +99,129 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) {
       protected[sql] def convertCTAS: Boolean =
         getConf("spark.sql.hive.convertCTAS", "false").toBoolean
     
    +  /**
    +   * The version of the hive client that will be used to communicate with the metastore.
 Note that
    +   * this does not necessarily need to be the same version of Hive that is used internally
by
    +   * Spark SQL for execution.
    +   */
    +  protected[hive] def hiveMetastoreVersion: String =
    +    getConf(HIVE_METASTORE_VERSION, "0.13.1")
    +
    +  /**
    +   * The location of the jars that should be used to instantiate the HiveMetastoreClient.
 This
    +   * property can be one of three options:
    +   *  - a colon-separated list of jar files or directories for hive and hadoop.
    +   *  - builtin - attempt to discover the jars that were used to load Spark SQL and use
those. This
    +   *              option is only valid when using the execution version of Hive.
    +   *  - maven - download the correct version of hive on demand from maven.
    +   */
    +  protected[hive] def hiveMetastoreJars: String =
    +    getConf(HIVE_METASTORE_JARS, "builtin")
    +
       @transient
       protected[sql] lazy val substitutor = new VariableSubstitution()
     
    +
    +  /** A local instance of hive that is only used for execution. */
    +  protected[hive] lazy val localMetastore = {
    +    val temp = Utils.createTempDir()
    +    temp.delete()
    +    temp
    +  }
    +
    +  @transient
    +  protected[hive] lazy val executionConf = new HiveConf()
    +  executionConf.set(
    +    "javax.jdo.option.ConnectionURL", s"jdbc:derby:;databaseName=$localMetastore;create=true")
    +
    +  /** The version of hive used internally by Spark SQL. */
    +  lazy val hiveExecutionVersion: String = "0.13.1"
    --- End diff --
    
    Its lazy because it ends up getting invoked during the constructor before it can be initialized,
but you make a good point.  I've moved it to the HiveContext singleton and replaced other
usages.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message