systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [systemml] niketanpansare commented on issue #857: [SYSTEMML-2523] Update SystemML to Support Spark 2.3.0
Date Thu, 21 Mar 2019 21:40:26 GMT
niketanpansare commented on issue #857: [SYSTEMML-2523] Update SystemML to Support Spark 2.3.0
URL: https://github.com/apache/systemml/pull/857#issuecomment-475413637
 
 
   Interestingly, running a similar code with `1.2.0` jars in `spark-2.3.0../spark-shell`
succeeds, i.e. behaves similar to setup 5 rather than setup 6. Here is the Scala code used
for testing:
   
   ```
   val ml = new org.apache.sysml.api.mlcontext.MLContext(spark)
   System.out.println(ml.version())
   val df = spark.read.parquet("shake.parquet")
   df.show()
   df.createOrReplaceTempView("df")
   ```
   
   Based on the above experiments, here are my thoughts:
   1. We can continue to support older Spark 2.1 version and can get away with warning on
Spark 2.3 in the following setups:
   - Invoked without any Spark SQL code
   - Part of Scala/Java pipeline (for example: if invoked via spark-shell)
   - With PySpark if and only if we recommend our users to not provide any jars in the `driver-class-path
or jars` (see setup 5 and 6)
   2. If we are uncomfortable with the above restriction, we should consider merging this
PR.
   - Though I have validated that above Python code works with Spark 2.2.3 with a warning,
I did not run exhaustive testing to guarantee backward compatibility support for older Spark
2.1 and 2.2 (with the exception of warning).
      

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message