spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudhakar Thota (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-14442) Error: Cannot load main class from JAR file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar
Date Wed, 06 Apr 2016 21:29:25 GMT

    [ https://issues.apache.org/jira/browse/SPARK-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229161#comment-15229161
] 

Sudhakar Thota commented on SPARK-14442:
----------------------------------------

These are the details:

1. Error I am facing:  Cannot load main class from JAR file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar

[spark@appdev1 ~]$ /usr/iop/current/spark-client/bin/spark-submit --verbose --master yarn-cluster
 --jars /usr/iop/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar, /usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar,
/usr/iop/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar --files /usr/iop/current/spark-client/conf/hive-site.xml
--class "SimpleApp" /tmp/simple-project_2.10-1.0.jar 
Using properties file: /usr/iop/4.1.0.0/spark/conf/spark-defaults.conf
Adding default property: spark.history.kerberos.keytab=none
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.yarn.queue=default
Adding default property: spark.yarn.historyServer.address=appdev1.softlayer.com:18080
Adding default property: spark.yarn.applicationMaster.waitTries=10
Adding default property: spark.history.kerberos.principal=none
Adding default property: spark.yarn.submit.file.replication=3
Adding default property: spark.yarn.am.extraJavaOptions=-Diop.version={{iop_full_version}}
Adding default property: spark.yarn.containerLauncherMaxThreads=25
Adding default property: spark.yarn.max.executor.failures=3
Adding default property: spark.driver.extraJavaOptions=-Diop.version={{iop_full_version}}
Adding default property: spark.yarn.preserve.staging.files=false
Adding default property: spark.eventLog.dir=hdfs://appdev1.softlayer.com:8020/iop/apps/4.1.0.0/spark/logs/history-server
Adding default property: spark.history.ui.port=18080
Adding default property: spark.yarn.executor.memoryOverhead=384
Adding default property: spark.yarn.driver.memoryOverhead=384
Adding default property: spark.yarn.scheduler.heartbeat.interval-ms=5000
Adding default property: spark.yarn.jar=hdfs://appdev1.softlayer.com:8020/iop/apps/4.1.0.0/spark/jars/spark-assembly.jar
Error: Cannot load main class from JAR file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar,
Run with --help for usage help or --verbose for debug output
[spark@appdev1 ~]$ 

2. The source code trying to execute.

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql
import org.apache.spark.sql.hive



object SimpleApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)


//    println("************************************************************")
//    println("We are in SPARK CONTEXT")
//    println("************************************************************")

    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    import sqlContext.implicits._
//    println("************************************************************")
//    println("We are in SQL CONTEXT which is actually HiveContext ")
//    println("************************************************************")
    

    val hc = new org.apache.spark.sql.hive.HiveContext(sc)
//    println("************************************************************")
//    println("We are in Hive CONTEXT")
//    println("************************************************************")
//    //val df = hc.read.format("parquet").load("/Users/sudhakarthota/spark/examples/src/main/resources/users.parquet")
//    //  ("namescolors_08172015_4")
//
//    //df.write.saveAsTable("namescolors_12162015_1")
//    //df.show(5)
    hc.sql("drop table mydbs")
    hc.sql("show databases").write.saveAsTable("mydbs")
  }
}

3. Spark versions that I ran this code on 1.4.1. and 1.5.1

4. Similar JIRA, that the problem is resolved in:

https://issues.apache.org/jira/browse/SPARK-4298



> Error: Cannot load main class from JAR file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-14442
>                 URL: https://issues.apache.org/jira/browse/SPARK-14442
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 1.4.1, 1.5.1
>            Reporter: Sudhakar Thota
>            Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message