mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruv Kumar <dku...@ecs.umass.edu>
Subject Re: Test Execution Time
Date Tue, 16 Aug 2011 18:25:29 GMT
Thankfully the tests for Mahout-627 (complete patch coming today), are
taking 57 seconds to run.

On Tue, Aug 16, 2011 at 2:19 PM, Sean Owen <srowen@gmail.com> wrote:

> (I disabled it. I had already tried to squeeze down its size without making
> it fail. I will write a note on the back of my hand to make sure to enable
> it before, say, releasing again.)
>
> On Tue, Aug 16, 2011 at 7:16 PM, Jake Mannix <jake.mannix@gmail.com>
> wrote:
>
> > As long as nobody's changing the logic going into Lanczos, disabling it
> > should be fine.  But I agree with Grant, that we should have it running
> > somewhere (continuous integration?).
> >
> > I can try and squeeze it's size down, but you don't get very good
> > convergence
> > on a small matrix when using these iterative matrix multiplier-style
> > algorithms,
> > which makes it hard to see that it's still working as expected.
> >
> >  -jake
> >
> > On Wed, Aug 10, 2011 at 11:54 PM, Sean Owen <srowen@gmail.com> wrote:
> >
> > > That would reduce memory requirements unilaterally? no, don't think so.
> I
> > > have not run into memory problems.
> > > The issue here is execution time and it's a big problem indeed.
> > >
> > > Would anyone object to disabling this test for now? it's getting costly
> > in
> > > the dev cycle.
> > >
> > > On Thu, Aug 11, 2011 at 5:52 AM, Lance Norskog <goksron@gmail.com>
> > wrote:
> > >
> > > > Another aspect of testing Map/Reduce jobs is memory bounds. An M/R
> job
> > > > should be able to run in a fairly constant amount of ram per JVM from
> > > > beginning to end, to avoid blowing up late in the game. Is there any
> > > > harness around that would do this?
> > > >
> > > > Lance
> > > >
> > > > On Sun, Aug 7, 2011 at 9:04 PM, Ted Dunning <ted.dunning@gmail.com>
> > > wrote:
> > > > > We have used testNg at work for just this segregation of fast and
> > slow
> > > > tests and it works pretty well. It is also useful for stripping out
> > > > cascading failures. This means that integration tests can be executed
> > > only
> > > > if the underlying unit tests succeed. Another nice thing is that
> testNg
> > > says
> > > > how many tests were skipped.
> > > > >
> > > > > Junit has recently been adding similar capabilities but I haven't
> > kept
> > > > up.
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > On Aug 7, 2011, at 19:18, Grant Ingersoll <gsingers@apache.org>
> > wrote:
> > > > >
> > > > >> It would likely help if we could get them to run in parallel,
> > perhaps.
> > > >  Also, seems like TestNG might have some better features on paper for
> > > this
> > > > kind of stuff (I think you can annotate some things as "slow" or
> "fast"
> > > and
> > > > then choose to run them separately).  I haven't explored much yet in
> > this
> > > > way.  Has anyone else used TestNG?
> > > > >>
> > > > >> -Grant
> > > > >>
> > > > >> On Aug 7, 2011, at 9:12 PM, Ted Dunning wrote:
> > > > >>
> > > > >>> I really don't think that this distinction needs to be made.
 The
> > > > >>> distinction between unit and integration test is important
from a
> > > > technical
> > > > >>> point of view, but as an organizing principle the topic and
> target
> > of
> > > > the
> > > > >>> test is probably a better idea than whether the test is a
> > functional
> > > or
> > > > unit
> > > > >>> test or whether it has randomized initial conditions or whether
> it
> > > has
> > > > for
> > > > >>> loops in it.  Tests should be organized by what they test.
> > > > >>>
> > > > >>> On Sun, Aug 7, 2011 at 5:08 PM, Lance Norskog <goksron@gmail.com
> >
> > > > wrote:
> > > > >>>
> > > > >>>> Sure! Perhaps the long-running ones can move to a new
> 'regression'
> > > > >>>> area? examples/ is partly what these are, so examples/regression
> > > makes
> > > > >>>> sense.
> > > > >>>>
> > > > >>>> On Sun, Aug 7, 2011 at 11:11 AM, Sean Owen <srowen@gmail.com>
> > > wrote:
> > > > >>>>> This test is indeed by far the culprit. I already
reduced its
> > test
> > > > input
> > > > >>>>> size to hurry it up, but it's gone slow again.
> > > > >>>>>
> > > > >>>>> Lance, indeed, these are not all unit tests -- nobody
said they
> > > were.
> > > > The
> > > > >>>>> test is useful.
> > > > >>>>>
> > > > >>>>> I do suggest, however, we comment it out. Jake suggested
it
> coudl
> > > be
> > > > made
> > > > >>>>> faster but I don't think he followed up.
> > > > >>>>>
> > > > >>>>> Sean
> > > > >>>>>
> > > > >>>>> On Sun, Aug 7, 2011 at 12:13 AM, Lance Norskog <
> > goksron@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> Comment out DistributedLanczosWhatsit. Zing!
> > > > >>>>>>
> > > > >>>>>> A unit test takes a bit of code X and checks
that code path A
> > goes
> > > > >>>>>> "tick" and code path B goes "tock" and bogus
input C throws an
> > > > >>>>>> exception. There's no such thing as a "unit test"
that runs
> > twelve
> > > > M/R
> > > > >>>>>> jobs in a row.
> > > > >>>>>>
> > > > >>>>>> There's MRUnit, which seems trapped in the Hadoop
> > > > 0.20/0.21/0.22/0.23
> > > > >>>>>> morass. This is a squib about how to do unit
testing of
> mappers
> > > and
> > > > >>>>>> reducers with Mockito:
> > > > >>>>>>
> > > > >>>>>> http://nubetech.co/testing-hadoop-map-reduce-jobs
> > > > >>>>>>
> > > > >>>>>> What the Mahout jobs want is more of a regression
test, which
> > > would
> > > > >>>>>> have two purposes:
> > > > >>>>>> 1) does the whole orchestration still work, and
> > > > >>>>>> 2) does it still acquire the information it is
supposed to
> > > acquire?
> > > > >>>>>> 2a) this requires some amount of real data and
a "gold
> standard"
> > > > >>>>>> output to match against.
> > > > >>>>>>
> > > > >>>>>> On Sat, Aug 6, 2011 at 12:34 PM, Grant Ingersoll
<
> > > > gsingers@apache.org>
> > > > >>>>>> wrote:
> > > > >>>>>>> Granted, I'm on a slow machine, but our tests
take forever to
> > > run.
> > > >  On
> > > > >>>> an
> > > > >>>>>> 2 core MBP, it takes well over an hour to run
all the tests (I
> > did
> > > > just
> > > > >>>>>> order a new MBP, so it will be faster, but it
doesn't lend
> > itself
> > > to
> > > > a
> > > > >>>> good
> > > > >>>>>> OOTB experience for people)
> > > > >>>>>>>
> > > > >>>>>>> One idea would be to add in parallel test
execution in Maven.
> >  I
> > > > think
> > > > >>>>>> this requires Mvn 3, but I am not sure.  Another
is to take a
> > look
> > > > at
> > > > >>>> our
> > > > >>>>>> tests, especially the slow ones and see if we
can speed them
> up.
> > > > >>>>>>>
> > > > >>>>>>> When I try adding in parallel tests to Maven,
I get a bunch
> of
> > > > >>>> failures
> > > > >>>>>> in the tests.
> > > > >>>>>>>
> > > > >>>>>>> I was using:
> > > > >>>>>>> <plugin>
> > > > >>>>>>>      <groupId>org.apache.maven.plugins</groupId>
> > > > >>>>>>>      <artifactId>maven-surefire-plugin</artifactId>
> > > > >>>>>>>      <configuration>
> > > > >>>>>>>        <forkMode>once</forkMode>
> > > > >>>>>>>        <argLine>-Xms256m -Xmx512m</argLine>
> > > > >>>>>>>        <testFailureIgnore>false</testFailureIgnore>
> > > > >>>>>>>
> >  <redirectTestOutputToFile>true</redirectTestOutputToFile>
> > > > >>>>>>>        <parallel>classes</parallel>
> > > > >>>>>>>        <threadCount>5</threadCount>
> > > > >>>>>>>      </configuration>
> > > > >>>>>>>    </plugin>
> > > > >>>>>>>
> > > > >>>>>>> Anyone played around with this stuff?  I
suspect the failures
> > are
> > > > due
> > > > >>>> to
> > > > >>>>>> tests stomping on each other, but I am still
digging in.
> > > > >>>>>>>
> > > > >>>>>>> -Grant
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> Lance Norskog
> > > > >>>>>> goksron@gmail.com
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Lance Norskog
> > > > >>>> goksron@gmail.com
> > > > >>>>
> > > > >>
> > > > >> --------------------------------------------
> > > > >> Grant Ingersoll
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Lance Norskog
> > > > goksron@gmail.com
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message