spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: reference to dstream in package org.apache.spark.streaming which is not available
Date Sat, 23 Aug 2014 01:47:21 GMT
The real fix is that the spark sink suite does not really need to use to
the spark-streaming test jars. Removing that dependency altogether, and
submitting a PR.

TD


On Fri, Aug 22, 2014 at 6:34 PM, Tathagata Das <tathagata.das1565@gmail.com>
wrote:

> Figured it out. Fixing this ASAP.
>
> TD
>
>
> On Fri, Aug 22, 2014 at 5:51 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
>
>> Hey All,
>>
>> We can sort this out ASAP. Many of the Spark committers were at a company
>> offsite for the last 72 hours, so sorry that it is broken.
>>
>> - Patrick
>>
>>
>> On Fri, Aug 22, 2014 at 4:07 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com
>> > wrote:
>>
>> > Sean - I think only the ones in 1726 are enough. It is weird that any
>> > class that uses the test-jar actually requires the streaming jar to be
>> > added explicitly. Shouldn't maven take care of this?
>> >
>> > I posted some comments on the PR.
>> >
>> > --
>> >
>> > Thanks,
>> > Hari
>> >
>> >
>> >  Sean Owen <mailto:sowen@cloudera.com>
>> >> August 22, 2014 at 3:58 PM
>> >>
>> >> Yes, master hasn't compiled for me for a few days. It's fixed in:
>> >>
>> >> https://github.com/apache/spark/pull/1726
>> >> https://github.com/apache/spark/pull/2075
>> >>
>> >> Could a committer sort this out?
>> >>
>> >> Sean
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: dev-help@spark.apache.org
>> >>
>> >> Ted Yu <mailto:yuzhihong@gmail.com>
>> >> August 22, 2014 at 1:55 PM
>> >>
>> >> Hi,
>> >> Using the following command on (refreshed) master branch:
>> >> mvn clean package -DskipTests
>> >>
>> >> I got:
>> >>
>> >> constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
>> >> ---------------------------------------------------
>> >> java.lang.reflect.InvocationTargetException
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> >> sun.reflect.NativeMethodAccessorImpl.invoke(
>> >> NativeMethodAccessorImpl.java:57)
>> >> at
>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >> DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:606)
>> >> at
>> >> org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> launchEnhanced(Launcher.java:289)
>> >> at
>> >> org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> launch(Launcher.java:229)
>> >> at
>> >> org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> mainWithExitCode(Launcher.java:415)
>> >> at org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> main(Launcher.java:356)
>> >> Caused by: scala.reflect.internal.Types$TypeError: bad symbolic
>> >> reference.
>> >> A signature in TestSuiteBase.class refers to term dstream
>> >> in package org.apache.spark.streaming which is not available.
>> >> It may be completely missing from the current classpath, or the
>> version on
>> >> the classpath might be incompatible with the version used when
>> compiling
>> >> TestSuiteBase.class.
>> >> at
>> >> scala.reflect.internal.pickling.UnPickler$Scan.
>> >> toTypeError(UnPickler.scala:847)
>> >> at
>> >> scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(
>> >> UnPickler.scala:854)
>> >> at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
>> >> at
>> >>
>> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
>> >> Types.scala:4280)
>> >> at
>> >>
>> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
>> >> Types.scala:4280)
>> >> at
>> >> scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.
>> >> scala:70)
>> >> at scala.collection.immutable.List.forall(List.scala:84)
>> >> at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(
>> >> Types.scala:4280)
>> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
>> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
>> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
>> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
>> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
>> >> at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
>> >> at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
>> >> at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
>> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
>> >> at
>> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
>> >> ExtractAPI.scala:296)
>> >> at
>> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
>> >> ExtractAPI.scala:296)
>> >> at
>> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
>> >> TraversableLike.scala:251)
>> >> at
>> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
>> >> TraversableLike.scala:251)
>> >> at
>> >> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.
>> >> scala:33)
>> >> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>> >> at scala.collection.TraversableLike$class.flatMap(
>> >> TraversableLike.scala:251)
>> >> at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
>> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI.
>> >> scala:296)
>> >> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
>> >> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
>> >> at xsbt.Message$$anon$1.apply(Message.scala:8)
>> >> at xsbti.SafeLazy$$anonfun$apply$1.apply(SafeLazy.scala:8)
>> >> at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:20)
>> >> at xsbti.SafeLazy$Impl._t(SafeLazy.scala:18)
>> >> at xsbti.SafeLazy$Impl.get(SafeLazy.scala:24)
>> >> at
>> xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
>> >> at
>> xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>> >> at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:138)
>> >> at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:139)
>> >> at xsbt.API$ApiPhase.processScalaUnit(API.scala:54)
>> >> at xsbt.API$ApiPhase.processUnit(API.scala:38)
>> >> at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
>> >> at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
>> >> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>> >> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>> >> at xsbt.API$ApiPhase.run(API.scala:34)
>> >> at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1583)
>> >> at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1557)
>> >> at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553)
>> >> at scala.tools.nsc.Global$Run.compile(Global.scala:1662)
>> >> at xsbt.CachedCompiler0.run(CompilerInterface.scala:123)
>> >> at xsbt.CachedCompiler0.run(CompilerInterface.scala:99)
>> >> at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> >> sun.reflect.NativeMethodAccessorImpl.invoke(
>> >> NativeMethodAccessorImpl.java:57)
>> >> at
>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >> DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:606)
>> >> at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:102)
>> >> at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:48)
>> >> at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)
>> >> at
>> >> sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.
>> >> apply$mcV$sp(AggressiveCompile.scala:99)
>> >> at
>> >> sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.
>> >> apply(AggressiveCompile.scala:99)
>> >> at
>> >> sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.
>> >> apply(AggressiveCompile.scala:99)
>> >> at
>> >> sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$
>> >> timed(AggressiveCompile.scala:166)
>> >> at
>> >> sbt.compiler.AggressiveCompile$$anonfun$3.compileScala$1(
>> >> AggressiveCompile.scala:98)
>> >> at
>> >>
>> sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala:
>> >> 143)
>> >> at
>> >>
>> sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala:
>> >> 87)
>> >> at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(
>> >> Compile.scala:39)
>> >> at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(
>> >> Compile.scala:37)
>> >> at sbt.inc.IncrementalCommon.cycle(Incremental.scala:99)
>> >> at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:38)
>> >> at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:37)
>> >> at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:65)
>> >> at sbt.inc.Incremental$.compile(Incremental.scala:37)
>> >> at sbt.inc.IncrementalCompile$.apply(Compile.scala:27)
>> >> at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:157)
>> >> at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:71)
>> >> at com.typesafe.zinc.Compiler.compile(Compiler.scala:184)
>> >> at com.typesafe.zinc.Compiler.compile(Compiler.scala:164)
>> >> at
>> sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:92)
>> >> at
>> >> scala_maven.ScalaCompilerSupport.incrementalCompile(
>> >> ScalaCompilerSupport.java:303)
>> >> at scala_maven.ScalaCompilerSupport.compile(
>> >> ScalaCompilerSupport.java:119)
>> >> at scala_maven.ScalaCompilerSupport.doExecute(
>> >> ScalaCompilerSupport.java:99)
>> >> at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
>> >> at
>> scala_maven.ScalaTestCompileMojo.execute(ScalaTestCompileMojo.java:48)
>> >> at
>> >> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(
>> >> DefaultBuildPluginManager.java:106)
>> >> at
>> >> org.apache.maven.lifecycle.internal.MojoExecutor.execute(
>> >> MojoExecutor.java:208)
>> >> at
>> >> org.apache.maven.lifecycle.internal.MojoExecutor.execute(
>> >> MojoExecutor.java:153)
>> >> at
>> >> org.apache.maven.lifecycle.internal.MojoExecutor.execute(
>> >> MojoExecutor.java:145)
>> >> at
>> >>
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(
>> >> LifecycleModuleBuilder.java:84)
>> >> at
>> >>
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(
>> >> LifecycleModuleBuilder.java:59)
>> >> at
>> >>
>> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(
>> >> LifecycleStarter.java:183)
>> >> at
>> >> org.apache.maven.lifecycle.internal.LifecycleStarter.
>> >> execute(LifecycleStarter.java:161)
>> >> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317)
>> >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152)
>> >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555)
>> >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
>> >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
>> >> ... 8 more
>> >>
>> >> Has anyone seen similar error ?
>> >>
>> >> Cheers
>> >>
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message