Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
In-Reply-To: <25E14699-FF4D-4672-9AB9-88CFC82BE22E@gmail.com>
To: dev@systemml.incubator.apache.org
Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
From: "Berthold Reinwald" <reinwald@us.ibm.com>
Date: Sat, 22 Oct 2016 02:14:25 -0700
References: <CAGU5spfYbJH4W5EeyG55=d4bWtugeCZ_owXU4H4TMndznEz51g@mail.gmail.com>
 <OF55451618.467D8F12-ON00258051.00035F04-88258051.00037A06@notes.na.collabserv.com>
 <OF17D1984D.5184B266-ON00258051.0038B41F-88258051.003930A7@notes.na.collabserv.com>
 <CAO1aQR8j4Nah6gdskEvCQvrRQJ5WbWvOLpxbd7ekV9cyGC=jQg@mail.gmail.com>
 <CAGU5speeELma+q3CZG4=MtvAx+BX=y+LVg20F6ZREsDDYzPFaA@mail.gmail.com>
 <CAO1aQR8pCDZM+zt3rR6uS+YYZF3S9peki8OVPa7ELyXGN9qb4w@mail.gmail.com>
 <b658a21a-2ea5-ad8e-1218-c39276a21ee6@gmail.com>
 <OF87664F1C.D21BAA38-ON00258051.0074F96C-88258051.007795EF@notes.na.collabserv.com>
 <535854B8-606E-46A6-865C-BC24049F42BA@gmail.com>
 <CAOcw5OwgHEEAn+kQGbkH6-QZfK=QC6fSbmL4u4rreYcXTSibJg@mail.gmail.com>
 <OF5294AC54.A4BF347C-ON00258053.00
 <25E14699-FF4D-4672-9AB9-88CFC82BE22E@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="=_alternative 002ECE7D88258054_="
Message-Id: <OF97E9C45C.213AAC19-ON88258054.002E3BE1-88258054.0032C2C8@notes.na.collabserv.com>
archived-at: Sat, 22 Oct 2016 09:14:44 -0000

--=_alternative 002ECE7D88258054_=
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="US-ASCII"

-1.

Transformencode throws an unnecessary error if strings to not comply with=20
the field requirements specified in RFC 4180. Arvind has a fix on the way=20
which should be included in the release.=20

Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinwald@us.ibm.com


From:   dusenberrymw@gmail.com
To:     dev@systemml.incubator.apache.org
Date:   10/21/2016 11:25 AM
Subject:        Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)


Okay I found out that the error I was encountering occurred due to passing =

in a DataFrame with an explicit row index column ("=5F=5FINDEX") that conta=
in=20
incorrect row indices. Basically, I had taken a large DataFrame with the=20
row index column  and sampled from it, without updating the row indices.=20
Thus, I was effectively left with sparse row indices -- i.e. I may have=20
had rows 2, 18, 587, 398678, etc. The current DataFrame conversion code=20
appears to not yet be able to handle sparse row indices and thus threw an=20
exception. When I correctly re-indexed the sampled DataFrame with dense=20
row indices, everything worked as expected. Of course, our conversion code =

automatically adds row indices to a given DataFrame during conversion if=20
the user does not supply them explicitly. However, it can save a bit of=20
time on repeated usage if it is done explicitly in one prior batch job.

I don't think this should block this release, and we should instead think=20
about this for the next release.  I've created SYSTEMML-1053 to track this =

issue.

I'm running a few more tests, and then I'll respond today with a vote.=20

-Mike

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Oct 20, 2016, at 10:48 PM, Glenn Weidner <gweidner@us.ibm.com> wrote:
>=20
> Similar release-process steps executed successfully on Windows.
>=20
> Performance test suite for large data still running; reviewing of=20
available log files in-progress.
>=20
> Thanks,
> Glenn
>=20
> Nakul Jindal ---10/20/2016 02:43:06 PM---Basic sanity tests pasts on=20
MacOS following the process here: http://apache.github.io/incubator-syst
>=20
> From: Nakul Jindal <nakul02@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 10/20/2016 02:43 PM
> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
>=20
>=20
>=20
>=20
> Basic sanity tests pasts on MacOS following the process here:
>=20
http://apache.github.io/incubator-systemml/release-process.html#all-binarie=
s-execute

>=20
> (The in-memory jar was removed by [SYSTEMML-741])
>=20
> +1
>=20
> Nakul Jindal
>=20
>=20
> On Thu, Oct 20, 2016 at 12:18 PM, <dusenberrymw@gmail.com> wrote:
>=20
> > Okay I've been testing the release candidate on a large-scale problem, =

and
> > I'm currently running into a "java.lang.NegativeArraySizeException" in
> > the SparseBlockMCSR that I do not believe was present previously. I'm
> > currently investigating, and will post again soon.
> >
> > On another note, I successfully ran all of the Python tests on both=20
Python
> > 2.7 and 3.5.
> >
> > -Mike
> >
> > --
> >
> > Mike Dusenberry
> > GitHub: github.com/dusenberrymw
> > LinkedIn: linkedin.com/in/mikedusenberry
> >
> > Sent from my iPhone.
> >
> >
> > > On Oct 19, 2016, at 2:46 PM, Glenn Weidner <gweidner@us.ibm.com>=20
wrote:
> > >
> > > Yes - that is correct for test cases involving ID column for
> > DataFrameVectorFrameConversionTest, DataFrameVectorScriptTest,
> > MLContextTest. The four failures for MLContextFrameTest were slightly
> > different and involve similar fix as done for FrameConverterTest under
> > [SYSTEMML-568] where FrameRDDConverterUtils.csvToRowRDDused to
> > incorporate schema information when converting to JavaRDD<Row>.
> > >
> > > Thanks,
> > > Glenn
> > >
> > > Matthias Boehm ---10/19/2016 12:36:04 PM---Glenn, all these issues=20
were
> > only caused by wrong tests that used an invalid ID schema or populated
> > >
> > > From: Matthias Boehm <mboehm7@googlemail.com>
> > > To: dev@systemml.incubator.apache.org
> > > Date: 10/19/2016 12:36 PM
> > > Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> > >
> > >
> > >
> > >
> > > Glenn, all these issues were only caused by wrong tests that used an
> > > invalid ID schema or populated this column incorrectly, right? If=20
so,
> > > then I think it's fine to release. However, if we touch it anyway,=20
we
> > > should globally change the ID schema from double to long, which is=20
more
> > > intuitive when created by hand.
> > >
> > > Regards,
> > > Matthias
> > >
> > > On 10/19/2016 8:30 PM, Deron Eriksson wrote:
> > > > OK, so I think it's my understanding that for the 'src' release=20
for
> > rc3,
> > > > the pom is using Spark 1.4 and the test suite passes for Spark=20
1.4, so
> > this
> > > > issue being discussed regarding test cases on Spark 1.6 is not a
> > blocker
> > > > for this release since the 'src' release builds and all tests=20
pass.
> > > >
> > > > If this is not correct, could someone please correct me?
> > > >
> > > > Deron
> > > >
> > > >
> > > > On Wed, Oct 19, 2016 at 11:17 AM, Luciano Resende <
> > luckbr1975@gmail.com>
> > > > wrote:
> > > >
> > > >> if tests are consistently failing, then we should cancel the RC=20
and
> > either
> > > >> fix the test or mark it as @ignored.
> > > >>
> > > >> Intermittent fails might be ok, but it's a community decision.
> > > >>
> > > >> On Wed, Oct 19, 2016 at 10:50 AM, Deron Eriksson <
> > deroneriksson@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I believe that for an Apache release, our test suite is supposed =

to
> > pass
> > > >>> (although I'm pretty sure random test fails can be ignored).
> > > >>>
> > > >>> See 2.1 of Release Check List here:
> > > >>>=20
http://incubator.apache.org/guides/releasemanagement.html#check-list
> > > >>>
> > > >>> "2.1 Build is successful including automated tests.
> > > >>> The expanded source archive is expected to build and pass=20
tests."
> > > >>>
> > > >>> Luciano, do you happen to know if some test failures are=20
acceptable
> > since
> > > >>> our test suite is so enormous (6300+ tests)?
> > > >>>
> > > >>> Deron
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Oct 19, 2016 at 3:24 AM, Glenn Weidner=20
<gweidner@us.ibm.com>
> > > >>> wrote:
> > > >>>
> > > >>>> It's a nice-to-have but not a release blocker.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Glenn
> > > >>>>
> > > >>>> [image: Inactive hide details for Niketan Pansare---10/18/2016
> > 05:38:26
> > > >>>> PM---Glenn: Would you prefer to have https://github.com/apache/
]
> > > >> Niketan
> > > >>>> Pansare---10/18/2016 05:38:26 PM---Glenn: Would you prefer to=20
have
> > > >>>> https://github.com/apache/incubator-systemml/pull/269 in 0.11
> > releas
> > > >>>>
> > > >>>> From: Niketan Pansare/Almaden/IBM@IBMUS
> > > >>>> To: dev@systemml.incubator.apache.org
> > > >>>> Date: 10/18/2016 05:38 PM
> > > >>>> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> > > >>>> ------------------------------
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Glenn: Would you prefer to have
> > > >>>> *https://github.com/apache/incubator-systemml/pull/269*
> > > >>>> <https://github.com/apache/incubator-systemml/pull/269> in 0.11
> > > >> release
> > > >>> ?
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Niketan Pansare
> > > >>>> IBM Almaden Research Center
> > > >>>> E-mail: npansar At us.ibm.com
> > > >>>> *http://researcher.watson.ibm.com/researcher/view.php?
> > > >> person=3Dus-npansar*
> > > >>>> <http://researcher.watson.ibm.com/researcher/view.php?
> > > >> person=3Dus-npansar>
> > > >>>>
> > > >>>> Luciano Resende ---10/17/2016 09:06:30 PM---Please note the=20
minor
> > > >>>> correction on the RC tag name (the actual tag hash is correct):
> > > >>>>
> > > >>>> From: Luciano Resende <luckbr1975@gmail.com>
> > > >>>> To: dev@systemml.incubator.apache.org
> > > >>>> Date: 10/17/2016 09:06 PM
> > > >>>> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> > > >>>> ------------------------------
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Please note the minor correction on the RC tag name (the actual =

tag
> > > >> hash
> > > >>> is
> > > >>>> correct):
> > > >>>>
> > > >>>> The tag to be voted on is v0.11.0-incubating-rc3 (
> > > >>>> 1baebfde400134b3af6d373c254ee084a6d28cc3)
> > > >>>>
> > > >>>>
> > > >>>> And off course, my +1
> > > >>>>
> > > >>>>
> > > >>>> On Sat, Oct 15, 2016 at 12:27 PM, Luciano Resende <
> > > >> luckbr1975@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>>
> > > >>>>> Please vote on releasing the following candidate as Apache=20
SystemML
> > > >>>>> version 0.11.0-incubating !
> > > >>>>>
> > > >>>>> The vote is open for at least 72 hours and passes if a=20
majority of
> > at
> > > >>>>> least 3 +1 PMC votes are cast.
> > > >>>>>
> > > >>>>> [ ] +1 Release this package as Apache SystemML=20
0.11.0-incubating
> > > >>>>> [ ] -1 Do not release this package because ...
> > > >>>>>
> > > >>>>> To learn more about Apache SystemML, please see
> > > >>>>> *http://systemml.apache.org/* <http://systemml.apache.org/>
> > > >>>>>
> > > >>>>> The tag to be voted on is v0.11.0-incubating-rc1 (
> > > >>>>> 1baebfde400134b3af6d373c254ee084a6d28cc3)
> > > >>>>>
> > > >>>>> *https://github.com/apache/incubator-systemml/tree/1baebfde40*
> > > >>>> <https://github.com/apache/incubator-systemml/tree/1baebfde40>
> > > >>>>> 0134b3af6d373c254ee084a6d28cc3
> > > >>>>>
> > > >>>>> The release artifacts can be found at :
> > > >>>>>
> > > >>>>> *https://dist.apache.org/repos/dist/dev/incubator/systemml/0*
> > > >>>> <https://dist.apache.org/repos/dist/dev/incubator/systemml/0>.
> > > >>>>> 11.0-incubating-rc3/
> > > >>>>>
> > > >>>>> The maven release artifacts, including signatures, digests,=20
etc.
> > can
> > > >> be
> > > >>>>> found at:
> > > >>>>>
> > > >>>>>
> > > >>>> *https://repository.apache.org/content/repositories/
> > > >>> orgapachesystemml-1009/*
> > > >>>> <https://repository.apache.org/content/repositories/
> > > >>> orgapachesystemml-1009/>
> > > >>>>>
> > > >>>>>
> > > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > >>>>> =3D=3D Apache Incubator release policy =3D=3D
> > > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > >>>>> Please find below the guide to release management during
> > incubation:
> > > >>>>> *http://incubator.apache.org/guides/releasemanagement.html*
> > > >>>> <http://incubator.apache.org/guides/releasemanagement.html>
> > > >>>>>
> > > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > >>>>> =3D=3D How can I help test this release? =3D=3D
> > > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > >>>>> If you are a SystemML user, you can help us test this release=20
by
> > > >> taking
> > > >>>> an
> > > >>>>> existing Algorithm or workload and running on this release
> > candidate,
> > > >>>> then
> > > >>>>> reporting any regressions.
> > > >>>>>
> > > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> > > >>>>> =3D=3D What justifies a -1 vote for this release? =3D=3D
> > > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> > > >>>>> -1 votes should only occur for significant stop-ship bugs or=20
legal
> > > >>>>> related issues (e.g. wrong license, missing header files,=20
etc).
> > Minor
> > > >>>> bugs
> > > >>>>> or regressions should not block this release.
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>> Luciano Resende
> > > >>>> *http://twitter.com/lresende1975* <
http://twitter.com/lresende1975>
> > > >>>> *http://lresende.blogspot.com/* <http://lresende.blogspot.com/>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Luciano Resende
> > > >> http://twitter.com/lresende1975
> > > >> http://lresende.blogspot.com/
> > > >>
> > > >
> > >
> > >
> > >
> > >
> >
>=20
>=20
>=20


--=_alternative 002ECE7D88258054_=--