royale-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Zarzycki <piotrzarzyck...@gmail.com>
Subject Re: Unit Tests et. al.
Date Sun, 05 Nov 2017 21:39:02 GMT
Harbs,

You can leave it to me Maven part, but I have to admit that I will be able
to take look into that probably at the end of the upcoming week. Still need
to finish release of Installer and SDK. If you find the time please raise
for me issue on GitHub, unless you would like to touch Maven.

Thanks! Piotr

2017-11-05 22:36 GMT+01:00 Harbs <harbs.lists@gmail.com>:

> Some progress on the unit tests:
>
> I got the Core tests to run on the swf side when doing a full build. In
> order to do that I needed to move the running of the tests until after
> Basic finished building so the test app could be built.
>
> We’re going to need to do something similar with the Maven build.
>
> The Testing project is compiling and is usable (on a basic level) too.
> Tests should run with both a SWF and JS target. The SWF target uses
> FlexUnit, while the JS target uses Josh’s implementation.
>
> I also migrated the Core tests to use the new project and that is building
> as well.
>
> I still need to add the bits that Greg wrote and wire up some JS tests.
>
> Hopefully I’ll make some more progress tomorrow.
>
> Harbs
>
> > On Nov 5, 2017, at 8:17 AM, Alex Harui <aharui@adobe.com.INVALID> wrote:
> >
> > Those warnings do not seem to affect the success of the tests on SWF for
> > me.
> >
> > IMO, the big challenge is in capturing test results so it can be reported
> > in a CI dashboard.  Having a GUI display results in Flash or in a browser
> > is a great first step, but I believe going from there to getting Selenium
> > to report individual test failures may be trickier.
> >
> > Peter tried to write up Mustella a long time ago. It might need updating.
> > See:
> > https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
> >
> > The advantage of declarative testing languages is that you get to
> > constrain the kinds of things that a test can do, and abstract when the
> > test harness does it.  It is much harder to interrupt sequential lines of
> > ActionScript/JavaScript.
> >
> > HTH,
> > -Alex
> >
> > On 11/4/17, 3:13 PM, "Harbs" <harbs.lists@gmail.com> wrote:
> >
> >> It looks like FlexUnit is currently broken.
> >>
> >> I’m getting the following output when trying to run the FlexUnit tests
> on
> >> Core and Basic:
> >>
> >>   [mxmlc]
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in
> the
> >> SWC
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>   [mxmlc]
> >>   [mxmlc]
> >>   [mxmlc]
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper
> in
> >> the SWC
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>
> >> Has anyone looked at this yet?
> >>
> >> Harbs
> >>
> >>> On Nov 4, 2017, at 10:45 PM, Harbs <harbs.lists@gmail.com> wrote:
> >>>
> >>> Thanks! That looks very useful.
> >>>
> >>> I started work on a feature/testing branch. I’ll try to merge what you
> >>> did into what Josh did. I’m going to try to get the existing Core
> and/or
> >>> Basic tests working in both swf and node js output tomorrow. We’ll see
> >>> how well I do… ;-)
> >>>
> >>> We might need different configs for different testing environments, but
> >>> I’ll see if we can keep the divergence of the environs is minimal as
> >>> possible.
> >>>
> >>> Once I get the basics worked out, I’ll likely have an idea what others
> >>> can help with. Let’s see if we can whip the Royale tests into shape.
> :-)
> >>>
> >>> Harbs
> >>>
> >>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <greg.dove@gmail.com> wrote:
> >>>>
> >>>> Harbs, I don't know if it still works since changes to Royale, but I
> >>>> had
> >>>> something rudimentary for cross-target unit testing quite shortly
> after
> >>>> working on the reflection support last year, in fact that was my
> >>>> primary
> >>>> reason for choosing to focus on reflection at the time.
> >>>>
> >>>> It was visual output only (i.e. just to look at the test results
> >>>> output in
> >>>> the browser) and the goal was to get some compatibility with flexunit
> >>>> (so
> >>>> that the same swf-based flexunit tests that were running in the build
> >>>> -e..g
> >>>> BinaryData tests- could be run visually side by side between swf and
> >>>> js).
> >>>> I also added a new TestVariance metadata to account for known
> >>>> (expected?)
> >>>> variation between swf and js. This was needed to cover (for example)
> >>>> things
> >>>> like testing Reflection classes themselves, where the results can be
> >>>> different between the targets based on the 'native' base classes (e..g
> >>>> EventDispatcher inheritance chain).
> >>>>
> >>>> Some of that code might also be useful to harvest for what you are
> >>>> doing,
> >>>> I'm not sure...
> >>>>
> >>>> It's here:
> >>>>
> >>>> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgithub.
> >>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
> 2Fmanualtests%2FUnitTests&da
> >>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa7b1b5a7b34438794aed
> >>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
> DX9gXiLqUoFQizp6xrakUwxjl
> >>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
> >>>>
> >>>>
> >>>>
> >>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <harbs.lists@gmail.com>
wrote:
> >>>>
> >>>>> Awesome!
> >>>>>
> >>>>> I assume that this means I can code the code to the Royale repo
under
> >>>>> Apache 2. Right?
> >>>>>
> >>>>> Harbs
> >>>>>
> >>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <joshtynjala@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Please feel free to use my test runner. I'm happy to commit
it
> >>>>>> somewhere
> >>>>> to make it official Apache code, if necessary.
> >>>>>>
> >>>>>> - Josh
> >>>>>>
> >>>>>> On 2017-11-03 04:19, Harbs <harbs.lists@gmail.com> wrote:
> >>>>>>> One topic which keeps coming up is better test coverage
for Royale.
> >>>>>>>
> >>>>>>> I think this is becoming a critical issue for a few reasons:
> >>>>>>> 1. As we get close to version 1.0 it’s necessary
to have good
> test
> >>>>> coverage for confidence of quality and confidence that we don’t
> >>>>> introduce
> >>>>> recessive bugs.
> >>>>>>> 2. It’s really hard to accept Github pull requests
without
> >>>>>>> examining
> >>>>> the code VERY well that it does not introduce recessive bugs. CI
> >>>>> which runs
> >>>>> automated tests could give a preliminary test on pull requests to
> >>>>> ensure
> >>>>> that they don’t break anything. If the pull requests do break
> >>>>> something,
> >>>>> it allows the requester to fix the problem with confidence without
> >>>>> taking
> >>>>> others’ time.
> >>>>>>>
> >>>>>>> I think we need to break up testing into pieces and figure
out a
> >>>>> strategy to implement automated tests in a way that they are
> >>>>> maintainable.
> >>>>>>>
> >>>>>>> Some points:
> >>>>>>> 1. I think integration into something like Travis would
be very
> >>>>>>> helpful.
> >>>>>>> 2. I think there’s a Jenkins plugin for building
pull requests.
> >>>>>>> Not
> >>>>> sure how useful it is.[1]
> >>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
> >>>>>>> which
> >>>>> could be useful for unit tests on parts of the framework which
> >>>>> don’t rely
> >>>>> on browser features. (i.e. models and the like) He mentioned the
> >>>>> possibility of donating it, and I think that would be a very useful
> >>>>> feature.
> >>>>>>> 4. For UI and integration tests, there seem to be some pretty
cool
> >>>>> integrations using Selenium.[2][3]
> >>>>>>> 5. I think the main testing effort needs to be using JS
and
> >>>>>>> something
> >>>>> like Josh’s testing framework for non-UI pieces and some
> easy-to-use
> >>>>> Selenium integration for visual pieces and integration tests.
> >>>>>>> 6. We probably also want some API endpoints we can test
off of for
> >>>>> integration testing.
> >>>>>>>
> >>>>>>> I’m willing to invest time into this.
> >>>>>>>
> >>>>>>> It’s going to be a lot of work building out the tests
and I think
> >>>>>>> we
> >>>>> need a plan for that. My thoughts:
> >>>>>>>
> >>>>>>> 1. Step one is to make it easy to write meaningful automated
tests
> >>>>>>> and
> >>>>> establish a clear pattern.
> >>>>>>> 2. Step two is to start writing tests starting from the
> >>>>> most-used/easiest to beak pieces and work out from there.
> >>>>>>> 3. Once the pattern is established, any new pieces MUST
have
> testing
> >>>>> coverage.
> >>>>>>> 4. When fixing bugs, attention should be paid to adding
testing for
> >>>>> that component.
> >>>>>>> 5. When a pull request comes in on a piece which does not
have unit
> >>>>> test, a test must be written before accepting the pull request.
The
> >>>>> test
> >>>>> does not need to be written by the requester. Before examining the
> >>>>> request,
> >>>>> the test should be written to pass for expected behavior and fail
for
> >>>>> the
> >>>>> bug that the pull request is attempting to fix (assuming the pull
> >>>>> request
> >>>>> is to fix a bug).
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>> Harbs
> >>>>>>>
> >>>>>>> P.S. I’m thinking of coming to the US in late December/early
> >>>>>>> January.
> >>>>> I would be interested in getting together for a hacking session
with
> >>>>> folks
> >>>>> who are available.
> >>>>>>>
> >>>>>>>
> >>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fw
> >>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
> data=02%7C01%7C%7C
> >>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7
> >>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
> 0uSl3
> >>>>>>> f0ezbSZY%3D&reserved=0+
> >>>>> request+builder+plugin
> >>>>> <https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fwiki.
> >>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa7b
> >>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> sdata=eqe6ec%
> >>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
> >>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
> >>>>>>>
> >>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fd
> >>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%
> 2F&data=02%7C01%
> >>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
> 7Cfa7b1b5a7b34438794aed2c178dec
> >>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
> 2FdOBmWJUsF7SylIQtMQQpAeOhw
> >>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
> >>>>>
> >>>>> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fdocs.t
> >>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
> 7C01%7C%7C67
> >>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7
> >>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
> jOytutB4
> >>>>> rGCY%3D&reserved=0>
> >>>>>>>
> >>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fs
> >>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%
> 7C%7C67a58a332b69
> >>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7C0%7C636
> >>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
> 2FNXSRVOQho9qUW1mqdw%3
> >>>>>>> D&reserved=0
> >>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fsau
> >>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa
> >>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> sdata=xpG
> >>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
> >>>>> products/integrations>
> >>>>>
> >>>>>
> >>>
> >>
> >
>
>


-- 

Piotr Zarzycki

Patreon: *https://www.patreon.com/piotrzarzycki
<https://www.patreon.com/piotrzarzycki>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message