systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dusenberr...@gmail.com
Subject Re: Time To Merge Spark 2.0 Support PR
Date Mon, 09 Jan 2017 20:28:12 GMT
Right, so we can cut a 0.12 release branch now, and then release from that, while work moves
forward on the master branch, including 2.0 support.

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jan 9, 2017, at 12:26 PM, Acs S <acs_s@yahoo.com.INVALID> wrote:
> 
> As I already mentioned we need to have proper way for Spark 1.6 users to use SystemML
Python DSL. Pip Install Artifact was missing from SystemML 0.11 release and it needs to be
added in SystemML 0.12 release.
> Arvind SurveSpark Technology Centerhttp://www.spark.tc
>      From: "dusenberrymw@gmail.com" <dusenberrymw@gmail.com>
> To: dev@systemml.incubator.apache.org 
> Sent: Monday, January 9, 2017 12:18 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
> Just to be clear, instead of creating a branch to merge the 2.0 support, we will want
to merge the 2.0 support into the master branch.
> 
> --
> 
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
> 
> Sent from my iPhone.
> 
> 
>> On Jan 9, 2017, at 12:02 PM, Acs S <acs_s@yahoo.com.INVALID> wrote:
>> 
>> Based on discussion thread we will start creating SystemML release based on Spark
2.0.
>> There are bunch of activities need to be completed and we need volunteer for most
of them.
>> Activity                                                                        
                                                                       Volunteer1. Create
a branch based on SystemML 0.12 release to merge Spark 2.0 code                          
     Luciano2. Get Spark 2.0 PR merged to this new branch.                               
                                                Glenn3. Do build changes to have both Spark
1.6 and 2.0 builds for release and PR.                          (Someone needs to work with
Alan)
>> 4. Setup Spark 2.0 cluster (One of the Almaden cluster updated with Spark 2.0)5.
Create Release Candidate                                                                 
                                            Glenn, Deron, Arvind6. Performance Testing7. Notebook
testing                                                                                  
                                           Arvind8. Python DSL verification (2.x and 3.x)9.
Scala DSL verification10. Artifacts verification11. Documentation update.
>> 
>> -----------------Arvind SurveSpark Technology Centerhttp://www.spark.tc
>> 
>>       From: Niketan Pansare <npansar@us.ibm.com>
>> To: dev@systemml.incubator.apache.org 
>> Sent: Friday, January 6, 2017 1:12 PM
>> Subject: Re: Time To Merge Spark 2.0 Support PR
>> 
>> I am fine with creating a branch for Spark 1.6 support and merging Spark 2.0 PR then.
Like Luciano said, we can creating a release 0.12 from our Spark 1.6 branch. 
>> 
>> Overriding previous release is common practice for pip installer, however pypi does
maintain the history of releases. Once a release candidate 0.12 is created, the user can install
SystemML python package in three ways:
>> 1. From source by checking out the branch and executing: mvn package -P distribution,
followed by pip install target/systemml-0.12.0-incubating-python.tgz
>> 2. From Apache site, pip install http://www.apache.org/dyn/closer.lua/incubator/systemml/0.12.0-incubating/systemml-0.12.0-incubating-python.tgz
>> 3. From pypi by specifying the version, pip install -I systemml_incubating==0.12
>> 
>> As long as we ensure that version of the python package on pypi matches our release
version and we document the Spark support in our release notes, there should not be any confusion
on usage :)
>> 
>> Thanks,
>> 
>> Niketan Pansare
>> IBM Almaden Research Center
>> E-mail: npansar At us.ibm.com
>> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>> 
>> Acs S ---01/06/2017 12:57:53 PM---I would agree to create a branch and add Spark
2.0 to it, while still releasing SystemML 0.12 releas
>> 
>> From: Acs S <acs_s@yahoo.com.INVALID>
>> To: "dev@systemml.incubator.apache.org" <dev@systemml.incubator.apache.org>
>> Date: 01/06/2017 12:57 PM
>> Subject: Re: Time To Merge Spark 2.0 Support PR
>> 
>> 
>> 
>> I would agree to create a branch and add Spark 2.0 to it, while still releasing SystemML
0.12 release with Pip Install Artifact.
>> Regarding comment from Mike, that new SystemML release will update PyPy package.Shouldn't
it be tagged with version #? Otherwise every release will override previous one.Niketan, any
comments?
>> -Arvind
>> 
>>       From: Matthias Boehm <mboehm7@googlemail.com>
>> To: dev@systemml.incubator.apache.org 
>> Sent: Friday, January 6, 2017 12:52 PM
>> Subject: Re: Time To Merge Spark 2.0 Support PR
>>   
>> +1 on moving to Spark 2.x - I think we delayed this way too long now and 
>> there will always be some awesome feature that we'd want to support on 
>> older Spark versions too.
>> 
>> Regards,
>> Matthias
>> 
>>> On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
>>> Well to be fair, a user can still use the Python DSL with the SystemML 0.11
>>> release by using `pip install -e src/main/python`.  We just didn't place a
>>> separate Python binary on the release website.  Keep in mind as well that
>>> once we release the next release with Spark 2.x support, a Spark 1.6 will
>>> not be able to use `pip install systemml` anyway, as that PyPy package will
>>> have been updated to the latest Spark 2.0 release.
>>> 
>>> I'm concerned that we are moving too slowly as a project without a clear
>>> set of users holding us back on 1.6.  Thoughts?
>>> 
>>> 
>>> --
>>> 
>>> Michael W. Dusenberry
>>> GitHub: github.com/dusenberrymw
>>> LinkedIn: linkedin.com/in/mikedusenberry
>>> 
>>>> On Fri, Jan 6, 2017 at 12:33 PM, Acs S <acs_s@yahoo.com.invalid> wrote:
>>>> 
>>>> With SystemML 0.11 release we don't have Python DSL support on Spark
>>>> 1.6.To have Python DSL on Spark 1.6 support we have to release SystemML
>>>> with Pip Install artifact, which was planned for SystemML 0.12.
>>>> I understand Spark 2.0 has released, at the same time there are Spark 1.6
>>>> users who will not able to use SystemML with Python DSL if we simply move
>>>> to Spark 2.0 without this artifact as planned. Understand SystemML 0.12 has
>>>> been delayed due to holiday period, but lets not get panicked and jump the
>>>> hops.
>>>> -Arvind
>>>> 
>>>>       From: Deron Eriksson <deroneriksson@gmail.com>
>>>>   To: dev@systemml.incubator.apache.org
>>>> Cc: Acs S <acs_s@yahoo.com>
>>>>   Sent: Friday, January 6, 2017 12:21 PM
>>>>   Subject: Re: Time To Merge Spark 2.0 Support PR
>>>> 
>>>> I would prefer to move forward with Spark 2 now.
>>>> 
>>>> Deron
>>>> 
>>>> 
>>>> 
>>>> On Fri, Jan 6, 2017 at 12:15 PM, Mike Dusenberry <dusenberrymw@gmail.com>
>>>> wrote:
>>>> 
>>>>> I vote that we scratch that release, and move forward with adding Spark
>>>> 2.x
>>>>> support now, and then release on February 1st as discussed on another
>>>>> thread this week.  Otherwise, our PR board will continue to be backed
up
>>>>> without real benefit to the user.
>>>>> 
>>>>> - Mike
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Michael W. Dusenberry
>>>>> GitHub: github.com/dusenberrymw
>>>>> LinkedIn: linkedin.com/in/mikedusenberry
>>>>> 
>>>>>> On Fri, Jan 6, 2017 at 12:03 PM, Acs S <acs_s@yahoo.com.invalid>
wrote:
>>>>>> 
>>>>>> We are trying to get SystemML 0.12 release out, which slowed down
due
>>>> to
>>>>>> holiday period in last 2+ weeksFinal verification pending from Glen
>>>>>> (Niketan, Berthed) related to run wrappers related code with 80GB
data.
>>>>>> Once this get verified we should be in position to release 0.12 and
>>>> then
>>>>>> can plan to get PR related to Spark 2.0 merged.
>>>>>> -Arvind
>>>>>> 
>>>>>>       From: Mike Dusenberry <dusenberrymw@gmail.com>
>>>>>>   To: dev <dev@systemml.incubator.apache.org>
>>>>>>   Sent: Friday, January 6, 2017 11:54 AM
>>>>>>   Subject: Time To Merge Spark 2.0 Support PR
>>>>>> 
>>>>>> Hi to the SystemML community!
>>>>>> 
>>>>>> As you may know, SystemML currently only supports Spark 1.6, and
does
>>>> not
>>>>>> yet support Spark 2.x.  However, there is an open PR (
>>>>>> https://github.com/apache/incubator-systemml/pull/202) that replaces
>>>>> Spark
>>>>>> 1.6 support with Spark 2.x support.
>>>>>> 
>>>>>> Spark 2.0.0 was released on July 26, 2016 (
>>>>>> https://spark.apache.org/news/spark-2-0-0-released.html), and 2.1.0
is
>>>>> now
>>>>>> available (https://spark.apache.org/news/spark-2-1-0-released.html).
>>>>>> 
>>>>>> I think it is time to merge the above PR into the master branch of
the
>>>>>> project in order to move SystemML onto Spark 2.x.
>>>>>> 
>>>>>> Thoughts?  If no objections, I'd like to merge next week.
>>>>>> 
>>>>>> 
>>>>>> Cheers!
>>>>>> 
>>>>>> - Mike
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> Michael W. Dusenberry
>>>>>> GitHub: github.com/dusenberrymw
>>>>>> LinkedIn: linkedin.com/in/mikedusenberry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Deron Eriksson
>>>> Spark Technology Center
>>>> http://www.spark.tc/
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>>   
>> 
>> 
>> 
>> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message