spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-23776) pyspark-sql tests should display build instructions when components are missing
Date Mon, 26 Mar 2018 22:56:01 GMT

     [ https://issues.apache.org/jira/browse/SPARK-23776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-23776:
------------------------------------

    Assignee: Apache Spark

> pyspark-sql tests should display build instructions when components are missing
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-23776
>                 URL: https://issues.apache.org/jira/browse/SPARK-23776
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 2.3.0
>            Reporter: Bruce Robbins
>            Assignee: Apache Spark
>            Priority: Minor
>
> This is a follow up to SPARK-23417.
> The pyspark-streaming tests print useful build instructions when certain components
are missing in the build.
> pyspark-sql's udf and readwrite tests also have specific build requirements: the build
must compile test scala files, and the build must also create the Hive assembly. When those
class or jar files are not created, the tests throw only partially helpful exceptions, e.g.:
> {noformat}
> AnalysisException: u'Can not load class test.org.apache.spark.sql.JavaStringLength, please
make sure it is on the classpath;'
> {noformat}
> or
> {noformat}
> IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':"
> {noformat}
> You end up in this situation when you follow Spark's build instructions and then attempt
to run the pyspark tests.
> It would be nice if pyspark-sql tests provide helpful build instructions in these cases.
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message