Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGU5spcXahshzrjv7rtY3VtrggOF3rN_bAY8_t233S3oHaWxmw@mail.gmail.com>
References: <OFF6FD1336.9AE1345C-ON00258003.00632818-88258003.0066F58B@notes.na.collabserv.com>
 <C9868F3C-214D-4820-AB38-D8B106C12F39@gmail.com> <CAO1aQR98AkCwTMH+qMBcUDHOp9dZXt+e2uX6-5ZjhkB4O6LWdA@mail.gmail.com>
 <OF17151879.3416D473-ONC2258004.007D9344-88258004.007DFED6@notes.na.collabserv.com>
 <OF5A43D06D.93CB4028-ON00258005.000120CD-88258005.000163AC@notes.na.collabserv.com>
 <OF759FC650.4D8BF62F-ON00258005.00287DF2-88258005.0029D496@notes.na.collabserv.com>
 <CAGU5spcXahshzrjv7rtY3VtrggOF3rN_bAY8_t233S3oHaWxmw@mail.gmail.com>
From: Deron Eriksson <deroneriksson@gmail.com>
Date: Fri, 5 Aug 2016 14:01:57 -0700
Message-ID: <CAO1aQR-N4EOpCT5zvPkhW7BmxTWZemYrmSxvPBDqO484imHoeQ@mail.gmail.com>
Subject: Re: [DISCUSS] Migration to Spark 2.0.0
To: dev@systemml.incubator.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0375588d91dd05395960f7
archived-at: Fri, 05 Aug 2016 21:02:04 -0000

--94eb2c0375588d91dd05395960f7
Content-Type: text/plain; charset=UTF-8

I am open to the idea of supporting Spark 2 and Spark<2 concurrently if
someone shows that it can be accomplished with minimal inconvenience.

However, I would lean towards Fred's approach (Spark 1.6 release followed
shortly by a Spark 2 release). If possible, I want to be able to focus most
of our efforts towards the future rather than the past.

Deron


On Thu, Aug 4, 2016 at 10:59 AM, Luciano Resende <luckbr1975@gmail.com>
wrote:

> That was going to be my suggestion... In Zeppelin, we just introduced
> support for different versions of scala and added support for spark 2.0
> based on profiles and a bit of reflections...
>
> Do we have to do anything related to Scala versions as well ?
>
> On Thursday, August 4, 2016, Matthias Boehm <mboehm@us.ibm.com> wrote:
>
> > I would recommend to start an investigation if we could support both the
> > 1.x and 2.x lines with a single code base. It seems feasible to refactor
> > the code a bit, compile against 2.0 (or with profiles), and run on either
> > 1.6 or 2.0. For example, by creating a wrapper that implements both
> > Iterable and Iterator, we could overcome the Iterator API change as shown
> > by our LazyIterableIterator which did not require any change in related
> > functions. Btw, we did the same for MRv1 and Yarn by ensuring that on
> MRv1,
> > we don't touch Yarn related APIs. Similarly on Spark, we already support
> > both legacy and >=1.6 memory management. I think this kind of platform
> > independence is really valuable but it obviously adds complexity.
> >
> > Regards,
> > Matthias
> >
> >
> > [image: Inactive hide details for Niketan Pansare---08/03/2016 05:15:21
> > PM---I am in favor of having one more release against Spark 1.6]Niketan
> > Pansare---08/03/2016 05:15:21 PM---I am in favor of having one more
> release
> > against Spark 1.6. Since default scala version for Spark 1.
> >
> > From: Niketan Pansare/Almaden/IBM@IBMUS
> > To: dev@systemml.incubator.apache.org
> > <javascript:_e(%7B%7D,'cvml','dev@systemml.incubator.apache.org');>
> > Date: 08/03/2016 05:15 PM
> > Subject: Re: [DISCUSS] Migration to Spark 2.0.0
> > ------------------------------
> >
> >
> >
> > I am in favor of having one more release against Spark 1.6. Since default
> > scala version for Spark 1.6 is 2.10, I recommend either having SystemML
> > compiled and released with Scala 2.10 profile or having two release
> > candidates.
> >
> > Thanks,
> >
> > Niketan Pansare
> > IBM Almaden Research Center
> > E-mail: npansar At us.ibm.com
> > *http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar*
> > <http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar>
> >
> > Frederick R Reiss---08/03/2016 03:58:17 PM---While I agree that getting
> > onto Spark 2.0 quickly ought to be a priority, there are existing early u
> >
> > From: Frederick R Reiss/Almaden/IBM@IBMUS
> > To: dev@systemml.incubator.apache.org
> > <javascript:_e(%7B%7D,'cvml','dev@systemml.incubator.apache.org');>
> > Date: 08/03/2016 03:58 PM
> > Subject: Re: [DISCUSS] Migration to Spark 2.0.0
> > ------------------------------
> >
> >
> >
> > While I agree that getting onto Spark 2.0 quickly ought to be a priority,
> > there are existing early users of SystemML who are likely stuck on Spark
> > 1.6.x for the next few months. Those users could want some of the new
> > experimental features since 0.10 (specifically frames, the prototype
> Python
> > DSL, and the new MLContext) and it would be good to have a Spark 1.6
> branch
> > of our version tree where we can backport the debugged versions of these
> > features if needed.
> >
> > I would recommend that we do one more SystemML release against Spark 1.6,
> > then switch the head version of SystemML over to Spark 2.0, then
> > immediately perform a second SystemML release. Thoughts?
> >
> > Fred
> >
> > Deron Eriksson ---08/02/2016 12:13:07 PM---I would definitely be in favor
> > of moving to Spark 2.0 as early as possible. This will allow SystemML
> >
> > From: Deron Eriksson <deroneriksson@gmail.com
> > <javascript:_e(%7B%7D,'cvml','deroneriksson@gmail.com');>>
> > To: dev@systemml.incubator.apache.org
> > <javascript:_e(%7B%7D,'cvml','dev@systemml.incubator.apache.org');>
> > Date: 08/02/2016 12:13 PM
> > Subject: Re: [DISCUSS] Migration to Spark 2.0.0
> > ------------------------------
> >
> >
> >
> > I would definitely be in favor of moving to Spark 2.0 as early as
> possible.
> > This will allow SystemML to be current with cutting edge Spark. It would
> be
> > nice to focus our efforts on the latest Spark.
> >
> > Deron
> >
> >
> > On Tue, Aug 2, 2016 at 12:05 PM, <dusenberrymw@gmail.com
> > <javascript:_e(%7B%7D,'cvml','dusenberrymw@gmail.com');>> wrote:
> >
> > > I'm in favor of moving to Spark 2.0 now, meaning that our upcoming
> > release
> > > would include both new features and 2.0 support.  0.10 has plenty of
> > > functionality for any existing 1.x users.
> > >
> > > -Mike
> > >
> > > --
> > >
> > > Mike Dusenberry
> > > GitHub: github.com/dusenberrymw
> > > LinkedIn: linkedin.com/in/mikedusenberry
> > >
> > > Sent from my iPhone.
> > >
> > >
> > > > On Aug 2, 2016, at 11:44 AM, Glenn Weidner <gweidner@us.ibm.com
> > <javascript:_e(%7B%7D,'cvml','gweidner@us.ibm.com');>> wrote:
> > > >
> > > >
> > > >
> > > > In the "[DISCUSS] SystemML 0.11 release" thread, native frame support
> > and
> > > > API updates such as new MLContext were identified as main new
> features
> > > for
> > > > the release.  In addition, support for Spark 2.0.0 was targeted.
> > > > Note code changes required for Spark 2.0.0 are not backward
> compatible
> > to
> > > > earlier Spark versions (e.g., 1.6.2) so starting separate mail thread
> > for
> > > > anyone to raise objections/alternatives for migrating to Spark 2.0.0.
> > > >
> > > > One possible option is to do a release to include the new Apache
> > SystemML
> > > > features before migrating to Spark 2.0.0.  However, it seems better
> to
> > > have
> > > > the next Apache SystemML release compatible with latest Spark version
> > > > 2.0.0.  The Apache SystemML 0.10 release from June can be used with
> > > earlier
> > > > versions of Spark.
> > > >
> > > > Regards,
> > > > Glenn
> > >
> >
> >
> >
> >
> >
> >
> >
>
> --
> Sent from my Mobile device
>

--94eb2c0375588d91dd05395960f7--