spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Invalid signature file digest for Manifest main attributes with spark job built using maven
Date Mon, 15 Sep 2014 22:37:44 GMT
This is more of a Java / Maven issue than Spark per se. I would use
the shade plugin to remove signature files in your final META-INF/
dir. As Spark does, in its <configuration>:

<filters>
  <filter>
    <artifact>*:*</artifact>
    <excludes>
      <exclude>org/datanucleus/**</exclude>
      <exclude>META-INF/*.SF</exclude>
      <exclude>META-INF/*.DSA</exclude>
      <exclude>META-INF/*.RSA</exclude>
    </excludes>
  </filter>
</filters>

On Mon, Sep 15, 2014 at 11:33 PM, kpeng1 <kpeng1@gmail.com> wrote:
> Hi All,
>
> I am trying to submit a spark job that I have built in maven using the
> following command:
> /usr/bin/spark-submit --deploy-mode client --class com.spark.TheMain
> --master local[1] /home/cloudera/myjar.jar 100
>
> But I seem to be getting the following error:
> Exception in thread "main" java.lang.SecurityException: Invalid signature
> file digest for Manifest main attributes
>         at
> sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
>         at
> sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
>         at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307)
>         at java.util.jar.JarVerifier.update(JarVerifier.java:218)
>         at java.util.jar.JarFile.initializeVerifier(JarFile.java:345)
>         at java.util.jar.JarFile.getInputStream(JarFile.java:412)
>         at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
>         at sun.misc.Resource.cachedInputStream(Resource.java:77)
>         at sun.misc.Resource.getByteBuffer(Resource.java:160)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:270)
>         at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:289)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Here is the pom file I am using to build the jar:
> <project xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd">
>   <modelVersion>4.0.0</modelVersion>
>   <groupId>com.spark</groupId>
>   <artifactId>myjar</artifactId>
>   <version>0.0.1-SNAPSHOT</version>
>   <name>${project.artifactId}</name>
>   <description>My wonderfull scala app</description>
>   <inceptionYear>2010</inceptionYear>
>   <licenses>
>     <license>
>       <name>My License</name>
>       <url>http://....</url>
>       <distribution>repo</distribution>
>     </license>
>   </licenses>
>
>   <properties>
>     <cdh.version>cdh5.1.0</cdh.version>
>     <maven.compiler.source>1.6</maven.compiler.source>
>     <maven.compiler.target>1.6</maven.compiler.target>
>     <encoding>UTF-8</encoding>
>     <scala.tools.version>2.10</scala.tools.version>
>     <scala.version>2.10.4</scala.version>
>   </properties>
>
>   <repositories>
>     <repository>
>       <id>scala-tools.org</id>
>       <name>Scala-tools Maven2 Repository</name>
>       <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
>     </repository>
>     <repository>
>       <id>maven-hadoop</id>
>       <name>Hadoop Releases</name>
>
> <url>https://repository.cloudera.com/content/repositories/releases/</url>
>     </repository>
>     <repository>
>       <id>cloudera-repos</id>
>       <name>Cloudera Repos</name>
>       <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
>     </repository>
>   </repositories>
>   <pluginRepositories>
>     <pluginRepository>
>       <id>scala-tools.org</id>
>       <name>Scala-tools Maven2 Repository</name>
>       <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
>     </pluginRepository>
>   </pluginRepositories>
>
>   <dependencies>
>     <dependency>
>       <groupId>org.scala-lang</groupId>
>       <artifactId>scala-library</artifactId>
>       <version>${scala.version}</version>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.spark</groupId>
>       <artifactId>spark-core_2.10</artifactId>
>       <version>1.0.0-${cdh.version}</version>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.spark</groupId>
>       <artifactId>spark-tools_2.10</artifactId>
>       <version>1.0.0-${cdh.version}</version>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.spark</groupId>
>       <artifactId>spark-streaming-flume_2.10</artifactId>
>       <version>1.0.0-${cdh.version}</version>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.spark</groupId>
>       <artifactId>spark-streaming_2.10</artifactId>
>       <version>1.0.0-${cdh.version}</version>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.flume</groupId>
>       <artifactId>flume-ng-sdk</artifactId>
>       <version>1.5.0-${cdh.version}</version>
>
>       <exclusions>
>         <exclusion>
>           <groupId>io.netty</groupId>
>           <artifactId>netty</artifactId>
>         </exclusion>
>       </exclusions>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.flume</groupId>
>       <artifactId>flume-ng-core</artifactId>
>       <version>1.5.0-${cdh.version}</version>
>
>       <exclusions>
>         <exclusion>
>           <groupId>io.netty</groupId>
>           <artifactId>netty</artifactId>
>         </exclusion>
>       </exclusions>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.hbase</groupId>
>       <artifactId>hbase-client</artifactId>
>       <version>0.98.1-${cdh.version}</version>
>
>       <exclusions>
>         <exclusion>
>           <groupId>io.netty</groupId>
>           <artifactId>netty</artifactId>
>         </exclusion>
>       </exclusions>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.hadoop</groupId>
>       <artifactId>hadoop-client</artifactId>
>       <version>2.3.0-${cdh.version}</version>
>
>     </dependency>
>
>
>     <dependency>
>       <groupId>junit</groupId>
>       <artifactId>junit</artifactId>
>       <version>4.11</version>
>       <scope>test</scope>
>     </dependency>
>     <dependency>
>       <groupId>org.specs2</groupId>
>       <artifactId>specs2_${scala.tools.version}</artifactId>
>       <version>1.13</version>
>       <scope>test</scope>
>     </dependency>
>     <dependency>
>       <groupId>org.scalatest</groupId>
>       <artifactId>scalatest_${scala.tools.version}</artifactId>
>       <version>2.0.M6-SNAP8</version>
>       <scope>test</scope>
>     </dependency>
>   </dependencies>
>
>   <build>
>     <sourceDirectory>src/main/scala</sourceDirectory>
>     <testSourceDirectory>src/test/scala</testSourceDirectory>
>     <plugins>
>       <plugin>
>
>         <groupId>net.alchim31.maven</groupId>
>         <artifactId>scala-maven-plugin</artifactId>
>         <version>3.1.3</version>
>         <executions>
>           <execution>
>             <goals>
>               <goal>compile</goal>
>               <goal>testCompile</goal>
>             </goals>
>             <configuration>
>               <args>
>                 <arg>-make:transitive</arg>
>                 <arg>-dependencyfile</arg>
>                 <arg>${project.build.directory}/.scala_dependencies</arg>
>               </args>
>             </configuration>
>           </execution>
>         </executions>
>       </plugin>
>       <plugin>
>         <groupId>org.apache.maven.plugins</groupId>
>         <artifactId>maven-surefire-plugin</artifactId>
>         <version>2.13</version>
>         <configuration>
>           <useFile>false</useFile>
>           <disableXmlReport>true</disableXmlReport>
>
>
>           <includes>
>             <include>**/*Test.*</include>
>             <include>**/*Suite.*</include>
>           </includes>
>         </configuration>
>       </plugin>
>       <plugin>
>         <groupId>org.apache.maven.plugins</groupId>
>         <artifactId>maven-shade-plugin</artifactId>
>         <version>2.2</version>
>         <configuration>
>
>           <transformers>
>             <transformer
> implementation="org.apache..plugins.shade.resource.ApacheLicenseResourceTransformer">
>             </transformer>
>             <transformer
> implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
>               <resource>reference.conf</resource>
>             </transformer>
>           </transformers>
>         </configuration>
>         <executions>
>           <execution>
>             <phase>package</phase>
>             <goals>
>               <goal>shade</goal>
>             </goals>
>           </execution>
>         </executions>
>       </plugin>
>     </plugins>
>   </build>
> </project>
>
> Was wondering what I am missing, since the jar built and a MANIFEST.MF file
> is created, not sure why it is looking for a signature file digest, am I
> missing some exclusion clause maybe?
>
> I saw a thread based on SBT and tried using the --deploy-mode client option;
> even though, my understanding is by default it would use that mode anyway.
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Invalid-signature-file-digest-for-Manifest-main-attributes-with-spark-job-built-using-maven-tp14299.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message