spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vanzin <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-11265] [YARN] YarnClient can't get toke...
Date Mon, 26 Oct 2015 16:11:50 GMT
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9232#discussion_r43014139
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
    @@ -142,6 +145,117 @@ class YarnSparkHadoopUtil extends SparkHadoopUtil {
         val containerIdString = System.getenv(ApplicationConstants.Environment.CONTAINER_ID.name())
         ConverterUtils.toContainerId(containerIdString)
       }
    +
    +  /**
    +   * Obtains token for the Hive metastore, using the current user as the principal.
    +   * Some exceptions are caught and downgraded to a log message.
    +   * @param conf hadoop configuration; the Hive configuration will be based on this
    +   * @return a token, or `None` if there's no need for a token (no metastore URI or principal
    +   *         in the config), or if a binding exception was caught and downgraded.
    +   */
    +  def obtainTokenForHiveMetastore(conf: Configuration): Option[Token[DelegationTokenIdentifier]]
= {
    +    try {
    +      obtainTokenForHiveMetastoreInner(conf, UserGroupInformation.getCurrentUser().getUserName)
    +    } catch {
    +      case e: Exception => {
    +        handleTokenIntrospectionFailure("Hive", e)
    +        None
    +      }
    +    }
    +  }
    +
    +  /**
    +   * Handle failures to obtain a token through introspection. Failures to load the class
are
    +   * not treated as errors: anything else is.
    +   * @param service service name for error messages
    +   * @param thrown exception caught
    +   * @throws Exception if the `thrown` exception isn't one that is to be ignored
    +   */
    +  private[yarn] def handleTokenIntrospectionFailure(service: String, thrown: Throwable):
Unit = {
    +    thrown match {
    +      case e: ClassNotFoundException =>
    +        logInfo(s"$service class not found $e")
    +        logDebug("Hive Class not found", e)
    +      case e: NoClassDefFoundError =>
    +        logDebug(s"$service class not found", e)
    +      case e: InvocationTargetException =>
    +        // problem talking to the metastore or other hive-side exception
    +        logInfo(s"$service method invocation failed", e)
    +        throw if (e.getCause != null) e.getCause else e
    +      case e: ReflectiveOperationException =>
    +        // any other reflection failure log at error and rethrow
    +        logError(s"$service Class operation failed", e)
    +        throw e;
    --- End diff --
    
    nuke the semi-colon. Also, I don't think you need to catch / log / re-throw any of these.
Just let them bubble up and fail the app. The user will see the error. As I suggested before,
the only `case` you need here is for `ClassNotFoundException`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message