systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dusenberr...@gmail.com
Subject Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
Date Fri, 21 Oct 2016 18:25:06 GMT
Okay I found out that the error I was encountering occurred due to passing in a DataFrame with
an explicit row index column ("__INDEX") that contain incorrect row indices. Basically, I
had taken a large DataFrame with the row index column  and sampled from it, without updating
the row indices. Thus, I was effectively left with sparse row indices -- i.e. I may have had
rows 2, 18, 587, 398678, etc. The current DataFrame conversion code appears to not yet be
able to handle sparse row indices and thus threw an exception. When I correctly re-indexed
the sampled DataFrame with dense row indices, everything worked as expected. Of course, our
conversion code automatically adds row indices to a given DataFrame during conversion if the
user does not supply them explicitly. However, it can save a bit of time on repeated usage
if it is done explicitly in one prior batch job.

I don't think this should block this release, and we should instead think about this for the
next release.  I've created SYSTEMML-1053 to track this issue.

I'm running a few more tests, and then I'll respond today with a vote. 

-Mike

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Oct 20, 2016, at 10:48 PM, Glenn Weidner <gweidner@us.ibm.com> wrote:
> 
> Similar release-process steps executed successfully on Windows.
> 
> Performance test suite for large data still running; reviewing of available log files
in-progress.
> 
> Thanks,
> Glenn
> 
> Nakul Jindal ---10/20/2016 02:43:06 PM---Basic sanity tests pasts on MacOS following
the process here: http://apache.github.io/incubator-syst
> 
> From: Nakul Jindal <nakul02@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 10/20/2016 02:43 PM
> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> 
> 
> 
> 
> Basic sanity tests pasts on MacOS following the process here:
> http://apache.github.io/incubator-systemml/release-process.html#all-binaries-execute
> 
> (The in-memory jar was removed by [SYSTEMML-741])
> 
> +1
> 
> Nakul Jindal
> 
> 
> On Thu, Oct 20, 2016 at 12:18 PM, <dusenberrymw@gmail.com> wrote:
> 
> > Okay I've been testing the release candidate on a large-scale problem, and
> > I'm currently running into a "java.lang.NegativeArraySizeException" in
> > the SparseBlockMCSR that I do not believe was present previously. I'm
> > currently investigating, and will post again soon.
> >
> > On another note, I successfully ran all of the Python tests on both Python
> > 2.7 and 3.5.
> >
> > -Mike
> >
> > --
> >
> > Mike Dusenberry
> > GitHub: github.com/dusenberrymw
> > LinkedIn: linkedin.com/in/mikedusenberry
> >
> > Sent from my iPhone.
> >
> >
> > > On Oct 19, 2016, at 2:46 PM, Glenn Weidner <gweidner@us.ibm.com> wrote:
> > >
> > > Yes - that is correct for test cases involving ID column for
> > DataFrameVectorFrameConversionTest, DataFrameVectorScriptTest,
> > MLContextTest. The four failures for MLContextFrameTest were slightly
> > different and involve similar fix as done for FrameConverterTest under
> > [SYSTEMML-568] where FrameRDDConverterUtils.csvToRowRDDused to
> > incorporate schema information when converting to JavaRDD<Row>.
> > >
> > > Thanks,
> > > Glenn
> > >
> > > Matthias Boehm ---10/19/2016 12:36:04 PM---Glenn, all these issues were
> > only caused by wrong tests that used an invalid ID schema or populated
> > >
> > > From: Matthias Boehm <mboehm7@googlemail.com>
> > > To: dev@systemml.incubator.apache.org
> > > Date: 10/19/2016 12:36 PM
> > > Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> > >
> > >
> > >
> > >
> > > Glenn, all these issues were only caused by wrong tests that used an
> > > invalid ID schema or populated this column incorrectly, right? If so,
> > > then I think it's fine to release. However, if we touch it anyway, we
> > > should globally change the ID schema from double to long, which is more
> > > intuitive when created by hand.
> > >
> > > Regards,
> > > Matthias
> > >
> > > On 10/19/2016 8:30 PM, Deron Eriksson wrote:
> > > > OK, so I think it's my understanding that for the 'src' release for
> > rc3,
> > > > the pom is using Spark 1.4 and the test suite passes for Spark 1.4, so
> > this
> > > > issue being discussed regarding test cases on Spark 1.6 is not a
> > blocker
> > > > for this release since the 'src' release builds and all tests pass.
> > > >
> > > > If this is not correct, could someone please correct me?
> > > >
> > > > Deron
> > > >
> > > >
> > > > On Wed, Oct 19, 2016 at 11:17 AM, Luciano Resende <
> > luckbr1975@gmail.com>
> > > > wrote:
> > > >
> > > >> if tests are consistently failing, then we should cancel the RC and
> > either
> > > >> fix the test or mark it as @ignored.
> > > >>
> > > >> Intermittent fails might be ok, but it's a community decision.
> > > >>
> > > >> On Wed, Oct 19, 2016 at 10:50 AM, Deron Eriksson <
> > deroneriksson@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I believe that for an Apache release, our test suite is supposed
to
> > pass
> > > >>> (although I'm pretty sure random test fails can be ignored).
> > > >>>
> > > >>> See 2.1 of Release Check List here:
> > > >>> http://incubator.apache.org/guides/releasemanagement.html#check-list
> > > >>>
> > > >>> "2.1 Build is successful including automated tests.
> > > >>> The expanded source archive is expected to build and pass tests."
> > > >>>
> > > >>> Luciano, do you happen to know if some test failures are acceptable
> > since
> > > >>> our test suite is so enormous (6300+ tests)?
> > > >>>
> > > >>> Deron
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Oct 19, 2016 at 3:24 AM, Glenn Weidner <gweidner@us.ibm.com>
> > > >>> wrote:
> > > >>>
> > > >>>> It's a nice-to-have but not a release blocker.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Glenn
> > > >>>>
> > > >>>> [image: Inactive hide details for Niketan Pansare---10/18/2016
> > 05:38:26
> > > >>>> PM---Glenn: Would you prefer to have https://github.com/apache/]
> > > >> Niketan
> > > >>>> Pansare---10/18/2016 05:38:26 PM---Glenn: Would you prefer
to have
> > > >>>> https://github.com/apache/incubator-systemml/pull/269 in 0.11
> > releas
> > > >>>>
> > > >>>> From: Niketan Pansare/Almaden/IBM@IBMUS
> > > >>>> To: dev@systemml.incubator.apache.org
> > > >>>> Date: 10/18/2016 05:38 PM
> > > >>>> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> > > >>>> ------------------------------
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Glenn: Would you prefer to have
> > > >>>> *https://github.com/apache/incubator-systemml/pull/269*
> > > >>>> <https://github.com/apache/incubator-systemml/pull/269>
in 0.11
> > > >> release
> > > >>> ?
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Niketan Pansare
> > > >>>> IBM Almaden Research Center
> > > >>>> E-mail: npansar At us.ibm.com
> > > >>>> *http://researcher.watson.ibm.com/researcher/view.php?
> > > >> person=us-npansar*
> > > >>>> <http://researcher.watson.ibm.com/researcher/view.php?
> > > >> person=us-npansar>
> > > >>>>
> > > >>>> Luciano Resende ---10/17/2016 09:06:30 PM---Please note the
minor
> > > >>>> correction on the RC tag name (the actual tag hash is correct):
> > > >>>>
> > > >>>> From: Luciano Resende <luckbr1975@gmail.com>
> > > >>>> To: dev@systemml.incubator.apache.org
> > > >>>> Date: 10/17/2016 09:06 PM
> > > >>>> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC3)
> > > >>>> ------------------------------
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Please note the minor correction on the RC tag name (the actual
tag
> > > >> hash
> > > >>> is
> > > >>>> correct):
> > > >>>>
> > > >>>> The tag to be voted on is v0.11.0-incubating-rc3 (
> > > >>>> 1baebfde400134b3af6d373c254ee084a6d28cc3)
> > > >>>>
> > > >>>>
> > > >>>> And off course, my +1
> > > >>>>
> > > >>>>
> > > >>>> On Sat, Oct 15, 2016 at 12:27 PM, Luciano Resende <
> > > >> luckbr1975@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>>
> > > >>>>> Please vote on releasing the following candidate as Apache
SystemML
> > > >>>>> version 0.11.0-incubating !
> > > >>>>>
> > > >>>>> The vote is open for at least 72 hours and passes if a
majority of
> > at
> > > >>>>> least 3 +1 PMC votes are cast.
> > > >>>>>
> > > >>>>> [ ] +1 Release this package as Apache SystemML 0.11.0-incubating
> > > >>>>> [ ] -1 Do not release this package because ...
> > > >>>>>
> > > >>>>> To learn more about Apache SystemML, please see
> > > >>>>> *http://systemml.apache.org/* <http://systemml.apache.org/>
> > > >>>>>
> > > >>>>> The tag to be voted on is v0.11.0-incubating-rc1 (
> > > >>>>> 1baebfde400134b3af6d373c254ee084a6d28cc3)
> > > >>>>>
> > > >>>>> *https://github.com/apache/incubator-systemml/tree/1baebfde40*
> > > >>>> <https://github.com/apache/incubator-systemml/tree/1baebfde40>
> > > >>>>> 0134b3af6d373c254ee084a6d28cc3
> > > >>>>>
> > > >>>>> The release artifacts can be found at :
> > > >>>>>
> > > >>>>> *https://dist.apache.org/repos/dist/dev/incubator/systemml/0*
> > > >>>> <https://dist.apache.org/repos/dist/dev/incubator/systemml/0>.
> > > >>>>> 11.0-incubating-rc3/
> > > >>>>>
> > > >>>>> The maven release artifacts, including signatures, digests,
etc.
> > can
> > > >> be
> > > >>>>> found at:
> > > >>>>>
> > > >>>>>
> > > >>>> *https://repository.apache.org/content/repositories/
> > > >>> orgapachesystemml-1009/*
> > > >>>> <https://repository.apache.org/content/repositories/
> > > >>> orgapachesystemml-1009/>
> > > >>>>>
> > > >>>>>
> > > >>>>> =====================================
> > > >>>>> == Apache Incubator release policy ==
> > > >>>>> =====================================
> > > >>>>> Please find below the guide to release management during
> > incubation:
> > > >>>>> *http://incubator.apache.org/guides/releasemanagement.html*
> > > >>>> <http://incubator.apache.org/guides/releasemanagement.html>
> > > >>>>>
> > > >>>>> =======================================
> > > >>>>> == How can I help test this release? ==
> > > >>>>> =======================================
> > > >>>>> If you are a SystemML user, you can help us test this
release by
> > > >> taking
> > > >>>> an
> > > >>>>> existing Algorithm or workload and running on this release
> > candidate,
> > > >>>> then
> > > >>>>> reporting any regressions.
> > > >>>>>
> > > >>>>> ================================================
> > > >>>>> == What justifies a -1 vote for this release? ==
> > > >>>>> ================================================
> > > >>>>> -1 votes should only occur for significant stop-ship bugs
or legal
> > > >>>>> related issues (e.g. wrong license, missing header files,
etc).
> > Minor
> > > >>>> bugs
> > > >>>>> or regressions should not block this release.
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>> Luciano Resende
> > > >>>> *http://twitter.com/lresende1975* <http://twitter.com/lresende1975>
> > > >>>> *http://lresende.blogspot.com/* <http://lresende.blogspot.com/>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Luciano Resende
> > > >> http://twitter.com/lresende1975
> > > >> http://lresende.blogspot.com/
> > > >>
> > > >
> > >
> > >
> > >
> > >
> >
> 
> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message