spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HyukjinKwon <...@git.apache.org>
Subject [GitHub] spark issue #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tests for ...
Date Sat, 03 Feb 2018 00:27:59 GMT
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20487
  
    Ah, nope. If PyArrow version is lower then we claim, for example, 0.7.0, seems tests go
failed:
    
    ```
    ======================================================================
    ERROR: test_vectorized_udf_wrong_return_type (pyspark.sql.tests.ScalarPandasUDF)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/.../spark/python/pyspark/sql/tests.py", line 4019, in test_vectorized_udf_wrong_return_type
        f = pandas_udf(lambda x: x * 1.0, MapType(LongType(), LongType()))
      File "/.../spark/python/pyspark/sql/functions.py", line 2309, in pandas_udf
        return _create_udf(f=f, returnType=return_type, evalType=eval_type)
      File "/.../spark/python/pyspark/sql/udf.py", line 47, in _create_udf
        require_minimum_pyarrow_version()
      File "/.../spark/python/pyspark/sql/utils.py", line 132, in require_minimum_pyarrow_version
        "however, your version was %s." % pyarrow.__version__)
    ImportError: pyarrow >= 0.8.0 must be installed on calling Python process; however,
your version was 0.7.0.
    
    ----------------------------------------------------------------------
    Ran 33 tests in 8.098s
    
    FAILED (errors=33)
    ```
    
    Will clarify it in PR description.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message