spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bruce Robbins (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23776) pyspark-sql tests should display build instructions when components are missing
Date Fri, 23 Mar 2018 03:00:03 GMT
Bruce Robbins created SPARK-23776:
-------------------------------------

             Summary: pyspark-sql tests should display build instructions when components
are missing
                 Key: SPARK-23776
                 URL: https://issues.apache.org/jira/browse/SPARK-23776
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 2.3.0
            Reporter: Bruce Robbins


This is a follow up to SPARK-23417.

The pyspark-streaming tests print useful build instructions when certain components are
missing in the build.

pyspark-sql's udf and readwrite tests also have specific build requirements: the build must
compile test scala files, and the build must also create the Hive assembly. When those class
or jar files are not created, the tests throw only partially helpful exceptions, e.g.:
{noformat}
AnalysisException: u'Can not load class test.org.apache.spark.sql.JavaStringLength, please
make sure it is on the classpath;'
{noformat}
or
{noformat}
IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':"
{noformat}
You end up in this situation when you follow Spark's build instructions and then attempt to
run the pyspark tests.

It would be nice if pyspark-sql tests provide helpful build instructions in these cases.
  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message