spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prashant Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.
Date Tue, 15 May 2018 10:49:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prashant Sharma updated SPARK-24279:
------------------------------------
    Description: 
Using test libraries available with spark sql streaming, produces weird incompatible byte
code errors. It is already tested on a different virtual box instance to make sure, that it
is not related to my system environment. A reproducer is uploaded to github.

[https://github.com/ScrapCodes/spark-bug-reproducer]

 

Doing a clean build reproduces the error.

 

Verbatim paste of the error.
{code:java}
[INFO] Compiling 1 source files to /home/prashant/work/test/target/test-classes at 1526380360990
[ERROR] error: missing or invalid dependency detected while loading class file 'QueryTest.class'.
[INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic
classpath.)
[INFO] A full rebuild may help if 'QueryTest.class' was compiled against an incompatible version
of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'.
[INFO] Could not access type PlanTestBase in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic
classpath.)
[INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against an incompatible
version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtils.class'.
[INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic
classpath.)
[INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against an incompatible
version of org.apache.spark.sql.catalyst.plans.
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25: error: Unable
to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product
types (case classes) are supported by importing spark.implicits._ Support for serializing
other types will be added in future releases.
[ERROR] val inputData = MemoryStream[Int]
[ERROR] ^
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30: error: Unable
to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product
types (case classes) are supported by importing spark.implicits._ Support for serializing
other types will be added in future releases.
[ERROR] CheckAnswer(2, 3, 4))
[ERROR] ^
[ERROR] 5 errors found
{code}
 

  was:
Using test libraries available with spark sql streaming, produces weird incompatible byte
code errors. It is already tested on a different virtual box instance to make sure, that it
is not related to my system environment. A reproducer is uploaded to github.

[https://github.com/ScrapCodes/spark-bug-reproducer]

 

Doing a clean build reproduces the error.


> Incompatible byte code errors, when using test-jar of spark sql.
> ----------------------------------------------------------------
>
>                 Key: SPARK-24279
>                 URL: https://issues.apache.org/jira/browse/SPARK-24279
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL, Structured Streaming
>    Affects Versions: 2.3.0, 2.4.0
>            Reporter: Prashant Sharma
>            Priority: Major
>
> Using test libraries available with spark sql streaming, produces weird incompatible
byte code errors. It is already tested on a different virtual box instance to make sure, that
it is not related to my system environment. A reproducer is uploaded to github.
> [https://github.com/ScrapCodes/spark-bug-reproducer]
>  
> Doing a clean build reproduces the error.
>  
> Verbatim paste of the error.
> {code:java}
> [INFO] Compiling 1 source files to /home/prashant/work/test/target/test-classes at 1526380360990
> [ERROR] error: missing or invalid dependency detected while loading class file 'QueryTest.class'.
> [INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the
problematic classpath.)
> [INFO] A full rebuild may help if 'QueryTest.class' was compiled against an incompatible
version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'.
> [INFO] Could not access type PlanTestBase in package org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the
problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against an incompatible
version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtils.class'.
> [INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the
problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against an incompatible
version of org.apache.spark.sql.catalyst.plans.
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25: error:
Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and
Product types (case classes) are supported by importing spark.implicits._ Support for serializing
other types will be added in future releases.
> [ERROR] val inputData = MemoryStream[Int]
> [ERROR] ^
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30: error:
Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and
Product types (case classes) are supported by importing spark.implicits._ Support for serializing
other types will be added in future releases.
> [ERROR] CheckAnswer(2, 3, 4))
> [ERROR] ^
> [ERROR] 5 errors found
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message