From dev-return-2635-archive-asf-public=cust-asf.ponee.io@systemml.apache.org Thu Mar 21 21:40:28 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D9B5518077A for ; Thu, 21 Mar 2019 22:40:27 +0100 (CET) Received: (qmail 74310 invoked by uid 500); 21 Mar 2019 21:40:27 -0000 Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.apache.org Delivered-To: mailing list dev@systemml.apache.org Received: (qmail 74288 invoked by uid 99); 21 Mar 2019 21:40:26 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Mar 2019 21:40:26 +0000 From: GitBox To: dev@systemml.apache.org Subject: [GitHub] [systemml] niketanpansare commented on issue #857: [SYSTEMML-2523] Update SystemML to Support Spark 2.3.0 Message-ID: <155320442645.26601.7799282749716394144.gitbox@gitbox.apache.org> Date: Thu, 21 Mar 2019 21:40:26 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit niketanpansare commented on issue #857: [SYSTEMML-2523] Update SystemML to Support Spark 2.3.0 URL: https://github.com/apache/systemml/pull/857#issuecomment-475413637 Interestingly, running a similar code with `1.2.0` jars in `spark-2.3.0../spark-shell` succeeds, i.e. behaves similar to setup 5 rather than setup 6. Here is the Scala code used for testing: ``` val ml = new org.apache.sysml.api.mlcontext.MLContext(spark) System.out.println(ml.version()) val df = spark.read.parquet("shake.parquet") df.show() df.createOrReplaceTempView("df") ``` Based on the above experiments, here are my thoughts: 1. We can continue to support older Spark 2.1 version and can get away with warning on Spark 2.3 in the following setups: - Invoked without any Spark SQL code - Part of Scala/Java pipeline (for example: if invoked via spark-shell) - With PySpark if and only if we recommend our users to not provide any jars in the `driver-class-path or jars` (see setup 5 and 6) 2. If we are uncomfortable with the above restriction, we should consider merging this PR. - Though I have validated that above Python code works with Spark 2.2.3 with a warning, I did not run exhaustive testing to guarantee backward compatibility support for older Spark 2.1 and 2.2 (with the exception of warning). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services