systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Berthold Reinwald" <>
Subject Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Date Thu, 18 Aug 2016 07:05:14 GMT
This makes sense. Couple of comments.

- wrt SystemML on Spark 1.x, SystemML 0.11 target date should be the 
latest in Oct. Ideally it should be earlier - maybe September - depending 
on community demand as it will contain bug fixes for features introduced 
in 0.10. The master branch will stay on Spark 1.x til then.  New features 
can be integrated but as they are still partial, they will be marked as 

- wrt SystemML on Spark 2.0, as the timeline is fairly short, instead of 
bi-monthly I'd suggest more frequent sync with the branch on Spark 2.0

- Comes early October, we should switch the master to Spark 2.0, and 
immediately create a release for 2.0.

- wrt Roadmap 0.11 items, except for Frame support and MLContext and may 
be DL lib, all the other features should be marked experimental. 

Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208

Date:   08/17/2016 04:52 PM
Subject:        Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap


This sounds like a good plan to allow us to continue supporting the Spark 
1.x line in the short term, with a plan for moving to Spark 2.x support 



Mike Dusenberry

Sent from my iPhone.

> On Aug 17, 2016, at 2:59 PM, Deron Eriksson <> 
> +1
> Continuing to support Spark 1.4/1.6 for now while setting a cutover date
> for 2.0 sounds like a great idea. This allows for the creation of a 
> solid release for 1.x, which greatly benefits SystemML users using Spark
> 1.x. It also gives these users a general date that they can use to plan
> migration to Spark 2.0 when that becomes the SystemML standard so that 
> can benefit from the latest improvements to the project.
> Deron
>> On Wed, Aug 17, 2016 at 2:32 PM, Acs S <> wrote:
>> Seems, mail is not retaining format. I am attaching same text through 
>> file.
>> If there is any other better option please let me know.
>> -Arvind
>> ----- Forwarded Message -----
>> *From:* Acs S <>
>> *To:* Dev <>
>> *Sent:* Wednesday, August 17, 2016 2:18 PM
>> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
>> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
>> uptodate with latest version of Spark. This brings us a challenge to
>> support our consumers until they move to Spark 2.0.Based on some
>> brainstorming, I can propose following options to keep SystemML being
>> supported on latest Spark version quickly.
>> Supporting SystemML on Spark 1.x We can continue to support SystemML on
>> Spark 1.x code base for short period of time by adding fixes and 
>> on main branch.  We will release SystemML with support to Spark 1.x 
>> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 
>> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
>> 2.0, we can make SystemML on Spark 2.0 immediately based on branch 
>> on top of latest master branch code. Glen has some prototype code to
>> transform SystemML code to be compatible with Spark 2.0, he can merge 
>> code with new branch targeted to support SystemML on Spark 2.0 This 
>> be "Preview" version code, and we can update it on frequent basis (on
>> bi-monthy basis).  Supporting SystemML on Spark 2.0 We will have full
>> support of SystemML on Spark 2.0 before end of year 2016. We will 
>> release date by end of Sept 2016. At the same time we will discuss if 
>> can move support of SystemML on Spark 1.x to maintenance mode (Only
>> required bug fixes will be merged from main branch) or we need to 
>> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
>> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
>> Learning (Library of Network layers?) - Frame - New MLContext API  -
>> Python DSL integration (Preview) - Compressed Linear Algebra (Preview) 
>> Hydra R integration - New Algorithms (?)
>> 0.12 (Spark 2.0)  (Targeted to 4Q 2016) - GPU support (Local
>> mode/Distributed mode?)
>> - New Algorithms (?)
>> Please feel free to comment on support and roadmap points.
>> -Arvind

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message