systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Boehm <mboe...@googlemail.com>
Subject Re: Time To Merge Spark 2.0 Support PR
Date Fri, 06 Jan 2017 20:52:48 GMT
+1 on moving to Spark 2.x - I think we delayed this way too long now and 
there will always be some awesome feature that we'd want to support on 
older Spark versions too.

Regards,
Matthias

On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
> Well to be fair, a user can still use the Python DSL with the SystemML 0.11
> release by using `pip install -e src/main/python`.  We just didn't place a
> separate Python binary on the release website.  Keep in mind as well that
> once we release the next release with Spark 2.x support, a Spark 1.6 will
> not be able to use `pip install systemml` anyway, as that PyPy package will
> have been updated to the latest Spark 2.0 release.
>
> I'm concerned that we are moving too slowly as a project without a clear
> set of users holding us back on 1.6.  Thoughts?
>
>
> --
>
> Michael W. Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> On Fri, Jan 6, 2017 at 12:33 PM, Acs S <acs_s@yahoo.com.invalid> wrote:
>
>> With SystemML 0.11 release we don't have Python DSL support on Spark
>> 1.6.To have Python DSL on Spark 1.6 support we have to release SystemML
>> with Pip Install artifact, which was planned for SystemML 0.12.
>> I understand Spark 2.0 has released, at the same time there are Spark 1.6
>> users who will not able to use SystemML with Python DSL if we simply move
>> to Spark 2.0 without this artifact as planned. Understand SystemML 0.12 has
>> been delayed due to holiday period, but lets not get panicked and jump the
>> hops.
>> -Arvind
>>
>>       From: Deron Eriksson <deroneriksson@gmail.com>
>>  To: dev@systemml.incubator.apache.org
>> Cc: Acs S <acs_s@yahoo.com>
>>  Sent: Friday, January 6, 2017 12:21 PM
>>  Subject: Re: Time To Merge Spark 2.0 Support PR
>>
>> I would prefer to move forward with Spark 2 now.
>>
>> Deron
>>
>>
>>
>> On Fri, Jan 6, 2017 at 12:15 PM, Mike Dusenberry <dusenberrymw@gmail.com>
>> wrote:
>>
>>> I vote that we scratch that release, and move forward with adding Spark
>> 2.x
>>> support now, and then release on February 1st as discussed on another
>>> thread this week.  Otherwise, our PR board will continue to be backed up
>>> without real benefit to the user.
>>>
>>> - Mike
>>>
>>>
>>> --
>>>
>>> Michael W. Dusenberry
>>> GitHub: github.com/dusenberrymw
>>> LinkedIn: linkedin.com/in/mikedusenberry
>>>
>>> On Fri, Jan 6, 2017 at 12:03 PM, Acs S <acs_s@yahoo.com.invalid> wrote:
>>>
>>>> We are trying to get SystemML 0.12 release out, which slowed down due
>> to
>>>> holiday period in last 2+ weeksFinal verification pending from Glen
>>>> (Niketan, Berthed) related to run wrappers related code with 80GB data.
>>>> Once this get verified we should be in position to release 0.12 and
>> then
>>>> can plan to get PR related to Spark 2.0 merged.
>>>> -Arvind
>>>>
>>>>      From: Mike Dusenberry <dusenberrymw@gmail.com>
>>>>  To: dev <dev@systemml.incubator.apache.org>
>>>>  Sent: Friday, January 6, 2017 11:54 AM
>>>>  Subject: Time To Merge Spark 2.0 Support PR
>>>>
>>>> Hi to the SystemML community!
>>>>
>>>> As you may know, SystemML currently only supports Spark 1.6, and does
>> not
>>>> yet support Spark 2.x.  However, there is an open PR (
>>>> https://github.com/apache/incubator-systemml/pull/202) that replaces
>>> Spark
>>>> 1.6 support with Spark 2.x support.
>>>>
>>>> Spark 2.0.0 was released on July 26, 2016 (
>>>> https://spark.apache.org/news/spark-2-0-0-released.html), and 2.1.0 is
>>> now
>>>> available (https://spark.apache.org/news/spark-2-1-0-released.html).
>>>>
>>>> I think it is time to merge the above PR into the master branch of the
>>>> project in order to move SystemML onto Spark 2.x.
>>>>
>>>> Thoughts?  If no objections, I'd like to merge next week.
>>>>
>>>>
>>>> Cheers!
>>>>
>>>> - Mike
>>>>
>>>> --
>>>>
>>>> Michael W. Dusenberry
>>>> GitHub: github.com/dusenberrymw
>>>> LinkedIn: linkedin.com/in/mikedusenberry
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Deron Eriksson
>> Spark Technology Center
>> http://www.spark.tc/
>>
>>
>>
>>
>

Mime
View raw message