spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Balog <doug.spark...@dugos.com>
Subject Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Date Thu, 22 Oct 2015 07:05:07 GMT

> On Oct 21, 2015, at 8:45 PM, Chester Chen <chester@alpinenow.com> wrote:
> 
> Doug
>   thanks for responding. 
>  >>I think Spark just needs to be compiled against 1.2.1
> 
>    Can you elaborate on this, or specific command you are referring ? 
> 
>    In our build.scala, I was including the following
> 
> "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" intransitive()
> 
>    I am not sure how the Spark compilation is directly related to this, please explain.
  

I was referring to this comment
https://issues.apache.org/jira/browse/SPARK-6906?focusedCommentId=14712336&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14712336
And the updated documentation,
http://spark.apache.org/docs/latest/sql-programming-guide.html#interacting-with-different-versions-of-hive-metastore

Perhaps I misunderstood your question and why you are trying to compile against a different
version of Hive.

> 
>    When we submit the spark job, the we call Spark Yarn Client.scala directly ( not using
spark-submit). 
>    The client side is not depending on spark-assembly jar ( which is in the hadoop cluster).
 The job submission actually failed in the client side. 
> 
>    Currently we get around this by replace the spark's hive-exec with apache hive-exec.

> 

Why are you using the Spark Yarn Client.scala directly and not using the SparkLauncher that
was introduced in 1.4.0 ?


Doug

> 
> 
> On Wed, Oct 21, 2015 at 5:27 PM, Doug Balog <doug@balog.net> wrote:
> See comments below.
> 
> > On Oct 21, 2015, at 5:33 PM, Chester Chen <chester@alpinenow.com> wrote:
> >
> > All,
> >
> >     just to see if this happens to other as well.
> >
> >   This is tested against the
> >
> >    spark 1.5.1 ( branch 1.5  with label 1.5.2-SNAPSHOT with commit on Tue Oct 6,
84f510c4fa06e43bd35e2dc8e1008d0590cbe266)
> >
> >    Spark deployment mode : Spark-Cluster
> >
> >    Notice that if we enable Kerberos mode, the spark yarn client fails with the
following:
> >
> > Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
> > java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$obtainTokenForHiveMetastore(Client.scala:1252)
> >         at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:271)
> >         at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:629)
> >         at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
> >         at org.apache.spark.deploy.yarn.Client.run(Client.scala:907)
> >
> >
> > Diving in Yarn Client.scala code and tested against different dependencies and notice
the followings:  if  the kerberos mode is enabled, Client.obtainTokenForHiveMetastore() will
try to use scala reflection to get Hive and HiveConf and method on these method.
> >
> >       val hiveClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.ql.metadata.Hive")
> >       val hive = hiveClass.getMethod("get").invoke(null)
> >
> >       val hiveConf = hiveClass.getMethod("getConf").invoke(hive)
> >       val hiveConfClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
> >
> >       val hiveConfGet = (param: String) => Option(hiveConfClass
> >         .getMethod("get", classOf[java.lang.String])
> >         .invoke(hiveConf, param))
> >
> >    If the "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" is used, then you
will get above exception. But if we use the
> >        "org.apache.hive" % "hive-exec" "0.13.1-cdh5.2.0"
> >  The above method will not throw exception.
> >
> >   Here some questions and comments
> > 0) is this a bug ?
> 
> I’m not an expert on this, but I think this might not be a bug.
> The Hive integration was redone for 1.5.0, see https://issues.apache.org/jira/browse/SPARK-6906
> and I think Spark just needs to be compiled against 1.2.1
> 
> 
> >
> > 1) Why spark-hive hive-exec behave differently ? I understand spark-hive hive-exec
has less dependencies
> >    but I would expect it functionally the same
> 
> I don’t know.
> 
> > 2) Where I can find the source code for spark-hive hive-exec ?
> 
> I don’t know.
> 
> >
> > 3) regarding the method obtainTokenForHiveMetastore(),
> >    I would assume that the method will first check if the hive-metastore uri is
present before
> >    trying to get the hive metastore tokens, it seems to invoke the reflection regardless
the hive service in the cluster is enabled or not.
> 
> Checking to see if the hive-megastore.uri is present before trying to get a delegation
token would be an improvement.
> Also checking to see if we are running in cluster mode would be good, too.
> I will file a JIRA and make these improvements.
> 
> > 4) Noticed the obtainTokenForHBase() in the same Class (Client.java) catches
> >    case e: java.lang.NoClassDefFoundError => logDebug("HBase Class not found:
" + e)
> >    and just ignore the exception ( log debug),
> >    but obtainTokenForHiveMetastore() does not catch NoClassDefFoundError exception,
I guess this is the problem.
> > private def obtainTokenForHiveMetastore(conf: Configuration, credentials: Credentials)
{
> >     // rest of code
> >  } catch {
> >     case e: java.lang.NoSuchMethodException => { logInfo("Hive Method not found
" + e); return }
> >     case e: java.lang.ClassNotFoundException => { logInfo("Hive Class not found
" + e); return }
> >     case e: Exception => { logError("Unexpected Exception " + e)
> >       throw new RuntimeException("Unexpected exception", e)
> >     }
> >   }
> > }
> 
> I tested the code against different scenarios, it possible that I missed the case where
the class was not found.
> obtainTokenForHBase() was implemented after obtainTokenForHive().
> 
> Cheers,
> 
> Doug
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message