Mailing-List: contact flex-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: flex-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of jeffxpx@gmail.com designates
 74.125.82.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CC4FE210.44809%aharui@adobe.com>
References: <CC5007E3.19D92%cframpto@adobe.com>
	<CC4FE210.44809%aharui@adobe.com>
Date: Tue, 14 Aug 2012 14:40:06 -0400
Message-ID: 
 <CAOs5KJjbDWCkO6sXe4JOdaKtZCzOeH2Dv8MP3akCKtQZOogtMA@mail.gmail.com>
Subject: Re: [RT] Ideas on getting the entire test suite for the sdk to run in
 10 minutes or less
From: Jeff Conrad <jeffxpx@gmail.com>
To: flex-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=f46d043892bd9ae69704c73e2324

--f46d043892bd9ae69704c73e2324
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Aug 14, 2012 at 2:08 PM, Alex Harui <aharui@adobe.com> wrote:

>
>
>
> On 8/14/12 10:56 AM, "Carol Frampton" <cframpto@adobe.com> wrote:
>
> >
> >
> > On 8/14/12 12 :01PM, "Jeff Conrad" <jeffxpx@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I'd like to help the project get to a point to where we can run the
> entire
> >> test suite for the sdk in 10 minutes or less.  I think that's a worthy
> >> goal, and I'm willing to help make that a reality.
> >>
> >> If we get the testing time down to being that fast, that will mean we
> can
> >> review a lot more contributions, and it could also mean that potential
> >> contributors can submit patches that they know don't fail any tests
> before
> >> submitting the patch.  It also means that if one of those tests fail,
> the
> >> person writing the code still has the context of what they did fresh in
> >> their mind and can fix it quickly.
> >>
> >> For reference, I ran the entire Mustella suite last night, and it took 4
> >> hours, 3 minutes, and 50 seconds to run ./mini_run.sh -createImages -all
> >> last night on a quad-core machine running Windows 7 using the Git Bash
> >> (yay! no cygwin!).  To make it work with the Git Bash, I changed the
> shell
> >> variable in mustella/build.xml to just sh.exe.  As a plus,
> shellrunner.sh
> >> somewhat intelligently parallel-ized the compilation of all the test
> swfs
> >> so it was compiling 4 swfs at a time, and then the ant script ran all
> the
> >> test swfs one at a time.  I know mustella is more a functional /
> >> integration test suite, so by definition it's going to run slower than a
> >> suite of unit tests.
> >>
> >> I also know that at least a few people on this list want to refactor the
> >> sdk so it's unit-testable.  I definitely support this and would like to
> >> help in any way I can on that front.
> >>
> >> I want the entire suite: unit, integration, and functional to be able to
> >> run in 10 minutes or less.  I have two ideas as to how we can make this
> >> happen, and I'm open to more.
> >>
> >> One idea would be to intelligently look at the files that a given patch
> /
> >> changeset affects and only build and run the tests that test that
> >> functionality.
> I have a prototype of selecting tests from an SVN Status report ready to
> go.
> I haven't tested it on a full set of files yet so it might need some
> tuning.
>

That's awesome!  So much better than starting from scratch.

>
> >
> > That's what we used to do at Adobe.  We had what we called a cyclone
> > server that we submitted a patch to and it ran the subset of tests based
> > on the changed files.  It used to take more than 10 minutes to get the
> > results though.
> I think Jeff is going to try to create a "massively" parallel system where
> we have a bundle of servers in the cloud each tasked with running a subset
> of the tests so they all get done in parallel.  There is generally no need
> for these tests to run serially.
>

Yep...my current thought right now is have the CI or something like it act
as a 'master' server any number of 'slave' servers, and have the master
give a slave a given subset to run and have it report back when done and
then give it another set to run until there are no more tests to run.  That
way we could spin up 2 or 200 and ideally it'd Just Work (tm).


>
> The only gotcha I've thought up so far is that the mustella tests use
> bitmap
> capture from the flashplayer.  That means the servers have to be running
> mac
> or windows because I don't think the linux player is on par with the other
> players.
>

I'll play around with a linux-based VM in vagrant and let you know.


> But I'm not up on cloud computing so maybe there is a way to do client-side
> testing in the cloud.
>

I have limited experience, but working stuff, running on Windows GUIs
automated using Windows Server 2003 R2.  Windows Server 2003 because it's a
legacy app that needs admin privileges for no particular reason, but I
didn't have source code...ugh...don't ask ;).  I had to play with the local
security policy settings or something like that and have it automatically
log in as the admin user so the desktop was always there, but it *does*
work.  The nice thing is flex and/or mustella will work fine as a normal
user, so I should be able to make it run on Windows Server 2008 without any
UAC-related troubles.

tl;dr Meme version:

I don't always run automated Windows GUI stuff in the cloud, but when I do,
I make it work :)


>
> > The problem with our implementation is the set of tests
> > that were run was based on a database that was manually put together, not
> > based on code inspection.  Over time the db became less and less accurate
> > because it wasn't kept up to date (and it might not ever have been
> totally
> > accurate because it was manually created).
> >
> > As to your other question about "hurricane".  I don't know what that is.
> > Maybe Alex remembers.  I was working on mustella/build.xml and
> mini_run.sh
> > last Friday.  I suspect there is a lot of dead code in there but we need
> > to get everything working first before we "clean".  The mustella
> directory
> > was previously maintained by the QE group for their setups.  I think
> there
> > are code paths in there that no longer apply.
> There is dead code in there that references the old QE DB.  I'm not sure if
> we'll need a similar DB or not.  I believe "hurricane" is another internal
> code word like "cyclone" that we used for qe/dev pair testing.
> >
> > BTW I'm not sure you can get the run down to 10 minutes but I think there
> > are an awful lot of duplicate tests and if we could figure which were
> > duplicates we could probably just throw away many tests.
> >
> > Carol
> >
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
>
>

--f46d043892bd9ae69704c73e2324--