cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Charbonneau <amcha...@gmail.com>
Subject Re: Cordova Screen Capture Plugin for Android
Date Sat, 16 Mar 2013 01:41:00 GMT
Ah thanks for clarifying that.

I'm not sure how I can alleviate the ui thread any more than I am
currently.  It is already doing the bare minimum amount of work on the
there.  Right now the only work done on the ui thread is
view.capturePicture() which is the recommended thread for doing that work
(otherwise you get a warning):


mUIThreadDone = false;
cordova.getActivity().runOnUiThread(new Runnable() {
         public void run() {
            	CordovaWebView uiThreadView = webView;
            	picture = uiThreadView.capturePicture();
            	mUIThreadDone = true;
            }
});
//rest of the work is done on core thread
while(!mUIThreadDone) {}
//write the picture to a file
...

I could do the file io and compares on a different thread than the
core one, but for the major use case, the core thread needs to wait no
matter what to ensure that the file is written and the comparison is
complete, so I wouldn't expect much gain there.


However, I've been experimenting with a more threaded version that
does the file io and comparison on a background thread as you
suggested.  If you have time please check it out. While the code is
not as clean, I think it addresses the issues you raised Andrew.  For
the major use case, performance is still the same as expected, but if
we want pure throughput on captures there's a big gain from about 3
captures per second (cps) to 10 cps on a Nexus 10.


Here's the new plugin, with moar threading!


https://github.com/Charbs09/Cordova-Mobile-Spec-ScreenCapture/blob/master/src/org/apache/cordova/plugin/ScreenCaptureMoarThreads.java


Once again, thanks for all your feedback on this, I haven't had much
experience with threading in java before now.


-Aaron





On Thu, Mar 14, 2013 at 8:10 AM, Andrew Grieve <agrieve@chromium.org> wrote:

> My suggestion was just to prevent the UI thread from locking up so that you
> app doesn't appear to be frozen when the plugin is doing its thing.
>
>
> On Wed, Mar 13, 2013 at 5:03 PM, Aaron Charbonneau <amcharbo@gmail.com
> >wrote:
>
> > Yah that's how I have it working now :) No problem with that approach for
> > working with Jasmine.
> >
> > I made reference to running them asynchronously to respond to Andrew
> about
> > returning immediately after the bits are captured, which in the case of
> > using that with Jasmine would only see a benefit IF we could run multiple
> > 'it' blocks at the same time.
> >
> > I think however Andrew was thinking of a different use case where we
> aren't
> > limited by a framework that requires tests to run serially, and results
> can
> > just come back as they are ready, and be evaluated all together at the
> end,
> > or not at all if you just want the captures and no evaluation.
> >
> > -Aaron
> >
> >
> > On Wed, Mar 13, 2013 at 1:53 PM, Braden Shepherdson <braden@chromium.org
> > >wrote:
> >
> > > As far as I know, you're correct: there's no way to move on the next
> it()
> > > block before the first one is done.
> > >
> > > What's the problem with just making the tests wait? Why does the next
> > it()
> > > block need to start early? You can make your call to Java, use waitFor
> to
> > > wait until the callback with the data is called, run your expectations,
> > and
> > > then move on to the next it() block. If the next it() block is using
> the
> > > result of this one, then they should be one it() block.
> > >
> > > Braden
> > >
> > >
> > > On Wed, Mar 13, 2013 at 4:46 PM, Aaron Charbonneau <amcharbo@gmail.com
> > > >wrote:
> > >
> > > > Hi Braden,
> > > > Sorry maybe I am overlooking something.  The way I understand it is
> > that
> > > a
> > > > previous 'it' block needs to complete before the next one can run.
>  So
> > I
> > > > have various tests setup like this:
> > > >
> > > > it("renders something", function() {
> > > > //draw something
> > > > ....
> > > > captureAndCompare(); //call the native function
> > > >  waitsFor(function() {
> > > > return captureComplete; //set by the native function callback
> > > > }, "capture never completed", 10000);
> > > > runs(function() {
> > > > expect(captureResult).toBe(0); //expects difference to be 0
> > > > });
> > > > });
> > > > //next 'it' test
> > > > ...
> > > >
> > > > The expect block needs to wait until there is a result returned from
> > the
> > > > native captureAndCompare call.  Therefore we use a waitsFor/runs
> block
> > > > there.
> > > >
> > > > But say we wanted to allow Javascript to run immediately after
> calling
> > > that
> > > > native function.  The only thing you need to wait for is that Java
> > > grabbed
> > > > the bits, and the rest of the processing will be done 'at some point
> in
> > > the
> > > > future.'  So as Andrew suggested you would return as soon as the bits
> > are
> > > > captured, and allow JS to run again.  The problem is that our expect
> > > block
> > > > needs the results, so that has to wait, the only other thing to do
> > would
> > > be
> > > > to run the next 'it' block.
> > > >
> > > > This is the part I didn't think was possible, can the next 'it' block
> > be
> > > > run while the previous 'it' is still waiting to execute the 'expect'
> > > block?
> > > >  Or is there another approach all together that would allow this?
> > > >
> > > > Thanks,
> > > > Aaron
> > > >
> > > >
> > > > On Wed, Mar 13, 2013 at 1:03 PM, Braden Shepherdson <
> > braden@chromium.org
> > > > >wrote:
> > > >
> > > > > Jasmine does support asynchronous tests, and waiting an arbitrary
> > > period
> > > > or
> > > > > until some condition is true. Why does that not work for these
> tests?
> > > > >
> > > > > Braden
> > > > >
> > > > >
> > > > > On Wed, Mar 13, 2013 at 3:42 PM, Aaron Charbonneau <
> > amcharbo@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Quick follow up question for the return values stuff:
> > > > > > Does it make sense to have my functions called from execute
> > > (capture()
> > > > > and
> > > > > > captureAndCompare()) return an error string if one is encountered
> > and
> > > > > pass
> > > > > > that back in callbackContext.error, then have execute return
> true?
> > > > > >
> > > > > > Or is there a preferred way to get errors from sub functions
back
> > > into
> > > > > > Javascript?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > >
> > > > > > On Wed, Mar 13, 2013 at 12:14 PM, Aaron Charbonneau <
> > > > amcharbo@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Thanks Andrew these are great suggestions!
> > > > > > >
> > > > > > > About not needing the busy-wait in getScreenBits:
> > > > > > > I'm not a fan of busy-waiting either.
> > > > > > > Perhaps it's a side effect of Jasmine requiring tests to
be run
> > and
> > > > > > > evaluated synchronously that I decided to use a busy-wait
in
> > > > > > getScreenBits.
> > > > > > >  If the Runnable calls CallbackContext.success/error once
the
> > bits
> > > > are
> > > > > > > captured (but not written to file) on the UIThread, the
> > Javascript
> > > > side
> > > > > > > will proceed to run before it has an image url/compare
result,
> > most
> > > > > > likely
> > > > > > > causing that test to fail.  As far as I know Jasmine doesn't
> > allow
> > > > for
> > > > > > > kicking off a bunch of tests and then gather all the results
at
> > the
> > > > end
> > > > > > and
> > > > > > > output pass/fails (each 'expect' function needs to be within
an
> > > 'it'
> > > > > > block,
> > > > > > > essentially that means one test must be evaluated before
the
> next
> > > can
> > > > > > > begin).
> > > > > > >
> > > > > > > I can see the usefulness of allowing the capture, file
io, and
> > > > > comparison
> > > > > > > to be able to run asynchronously from the Javascript though.
In
> > the
> > > > > case
> > > > > > > where you have your own test framework that CAN kick off
a
> bunch
> > of
> > > > > tests
> > > > > > > and just gather the results as they come in and do a batch
> > > evaluation
> > > > > at
> > > > > > > the end, or in the case where you don't want to do any
> evaluation
> > > in
> > > > > your
> > > > > > > app and just want to get captures as fast as you can,
> > asynchronous
> > > > > could
> > > > > > > increase performance there.
> > > > > > >
> > > > > > > I could try to implement it this way too, if you think
that's a
> > > worth
> > > > > > > while use case, maybe just provide an 'asynchronous' flag
in
> > > > > > CaptureOptions?
> > > > > > >
> > > > > > > For doing the fileIO/compare using cordova.getThreadPool,
is
> > there
> > > a
> > > > > > > benefit to doing that if I still plan having the operation
> block
> > > the
> > > > > > > Javascript?  For asynchronous mode I see why that makes
sense.
> > > > > > >
> > > > > > > For the return values, I'll make those changes right away!
> >  Thanks
> > > > > again
> > > > > > > for providing your insight, hopefully I understood it
> correctly.
> > > > > > >
> > > > > > > -Aaron
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Mar 13, 2013 at 8:04 AM, Andrew Grieve <
> > > agrieve@chromium.org
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Had a glance at your code. Things you may want to address:
> > > > > > >>
> > > > > > >> while(!mUIThreadDone) {}
> > > > > > >>
> > > > > > >> You shouldn't need to busy-wait. Pass the CallbackContext
to
> the
> > > > > > Runnable,
> > > > > > >> and call success/error on it whenever you're done.
No need to
> > have
> > > > the
> > > > > > web
> > > > > > >> thread wait for it. Even better would be to use a second
> > Runnable
> > > to
> > > > > do
> > > > > > >> the
> > > > > > >> image compare / file IO on a background thread via
> > > > > > cordova.getThreadPool()
> > > > > > >>
> > > > > > >> The return value to execute() should only be false
if an
> invalid
> > > > > command
> > > > > > >> was passed. It's not meant to indicate the success
of the
> > > operation.
> > > > > You
> > > > > > >> should be returning true for it.
> > > > > > >>
> > > > > > >> Instead of return "success"/"", it might simplify things
a bit
> > to
> > > > > > return a
> > > > > > >> boolean of true/false
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Mar 12, 2013 at 4:51 PM, Lorin Beer <
> > > > lorin.beer.dev@gmail.com
> > > > > > >> >wrote:
> > > > > > >>
> > > > > > >> > all the same, very impressive and faster then
I expected.
> > > > > > >> >
> > > > > > >> > - Lorin
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Mar 12, 2013 at 1:46 PM, Aaron Charbonneau
<
> > > > > > amcharbo@gmail.com
> > > > > > >> > >wrote:
> > > > > > >> >
> > > > > > >> > > Thanks Lorin.
> > > > > > >> > >
> > > > > > >> > > I did a quick test for throughput on an animating
canvas.
>  I
> > > was
> > > > > > able
> > > > > > >> to
> > > > > > >> > > grab 30 frames within a 5 seconds which equates
to around
> > > 166ms
> > > > > per
> > > > > > >> > capture
> > > > > > >> > > on a Galaxy Nexus.  Unfortunately not fast
enough to
> > automate
> > > > > > >> something
> > > > > > >> > > like video.  As with any kind of capturing
system, often
> the
> > > > > > overhead
> > > > > > >> of
> > > > > > >> > > copying bits around is enough to slow down
the operation
> of
> > > the
> > > > > app.
> > > > > > >> > >
> > > > > > >> > > -Aaron
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Tue, Mar 12, 2013 at 11:58 AM, Lorin Beer
<
> > > > > > >> lorin.beer.dev@gmail.com
> > > > > > >> > > >wrote:
> > > > > > >> > >
> > > > > > >> > > > Hey Aaron,
> > > > > > >> > > >
> > > > > > >> > > > very cool stuff, looking forward to
checking it out!
> > > > > > >> > > >
> > > > > > >> > > > Question: any performance hit on the
app by using
> Capture?
> > > > > What's
> > > > > > >> the
> > > > > > >> > > > expected throughput on images taken
in this way (given a
> > > > > > particular
> > > > > > >> > > device,
> > > > > > >> > > > say Galaxy Nexus)?
> > > > > > >> > > >
> > > > > > >> > > > - Lorin
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Mar 12, 2013 at 11:30 AM, Aaron
Charbonneau <
> > > > > > >> > amcharbo@gmail.com
> > > > > > >> > > > >wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Glad you like it :)
> > > > > > >> > > > > Yes in fact the actual capture
makes use of
> > > > > > View.capturePicture()
> > > > > > >> > which
> > > > > > >> > > > > actually grabs the entire document,
then that can be
> > > clipped
> > > > > > down
> > > > > > >> to
> > > > > > >> > > the
> > > > > > >> > > > > size/location of a specific element.
> > > > > > >> > > > >
> > > > > > >> > > > > -Aaron
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue, Mar 12, 2013 at 11:27 AM,
Michal Mocny <
> > > > > > >> mmocny@chromium.org>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Aaron,
> > > > > > >> > > > > >
> > > > > > >> > > > > > I haven't even begun looking
at your implementation,
> > but
> > > > I'm
> > > > > > >> just
> > > > > > >> > > going
> > > > > > >> > > > > to
> > > > > > >> > > > > > say it: this is awesome!
> > > > > > >> > > > > >
> > > > > > >> > > > > > First question: When Capturing
a DOM element, can
> you
> > > > > capture
> > > > > > >> > 'body'
> > > > > > >> > > to
> > > > > > >> > > > > > grab it&children for a
"full content screenshot", or
> > > does
> > > > it
> > > > > > >> have
> > > > > > >> > to
> > > > > > >> > > > be a
> > > > > > >> > > > > > specific single element?
> > > > > > >> > > > > >
> > > > > > >> > > > > > -Michal
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Tue, Mar 12, 2013 at 1:58
PM, Aaron Charbonneau <
> > > > > > >> > > amcharbo@gmail.com
> > > > > > >> > > > > > >wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Greetings!  My name is
Aaron Charbonneau, happy to
> > be
> > > a
> > > > > new
> > > > > > >> > member
> > > > > > >> > > of
> > > > > > >> > > > > the
> > > > > > >> > > > > > > mailing list!
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I have been developing
a sceenshot plugin for
> > Cordova
> > > to
> > > > > > help
> > > > > > >> > > > > facilitate
> > > > > > >> > > > > > > automation testing and
debugging of Cordova apps,
> > and
> > > I
> > > > > > would
> > > > > > >> > love
> > > > > > >> > > > some
> > > > > > >> > > > > > > feedback on it.  Currently
Cordova provides a
> bunch
> > of
> > > > > > native
> > > > > > >> > > > functions
> > > > > > >> > > > > > > that allow you to do
some cool stuff, but not much
> > > > > > >> functionality
> > > > > > >> > to
> > > > > > >> > > > > test
> > > > > > >> > > > > > > the apps that make use
of them.  Being able to
> take
> > a
> > > > > > capture
> > > > > > >> of
> > > > > > >> > > the
> > > > > > >> > > > > > screen
> > > > > > >> > > > > > > from within you app is
a great way to automate
> > testing
> > > > or
> > > > > > get
> > > > > > >> > > > > additional
> > > > > > >> > > > > > > information for debugging
an issue.  Since there
> is
> > no
> > > > > > >> > > > > > > Javascript mechanism
for taking screen captures
> the
> > > > > solution
> > > > > > >> > would
> > > > > > >> > > > have
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > be native, which fits
nicely into the "gap" that
> > > > > > >> Cordova/Phonegap
> > > > > > >> > > > > > bridges.
> > > > > > >> > > > > > >  Any medium to large
scale app can benefit greatly
> > > from
> > > > > > >> > automation
> > > > > > >> > > > > > testing
> > > > > > >> > > > > > > and any app can benefit
from an extra debugging
> > tool,
> > > > and
> > > > > > >> that is
> > > > > > >> > > > what
> > > > > > >> > > > > I
> > > > > > >> > > > > > > hope this screenshot
plugin can help achieve.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Currently the plugin
offers 2 functions:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Capture():
> > > > > > >> > > > > > > * Take a capture of the
current view, write that
> > > capture
> > > > > to
> > > > > > a
> > > > > > >> > .png
> > > > > > >> > > > file
> > > > > > >> > > > > > > with the specified file
name and sub directory of
> > the
> > > > > sdcard
> > > > > > >> > > > (fallback
> > > > > > >> > > > > to
> > > > > > >> > > > > > > emulated sdcard in the
case there isn't an sdcard
> > > > mounted)
> > > > > > >> > > > > > > * Able to create a sub-screenshot
with a specified
> > > > > rectangle
> > > > > > >> in
> > > > > > >> > > order
> > > > > > >> > > > > to
> > > > > > >> > > > > > > block out ui elements
that may be variable, and
> also
> > > > save
> > > > > > >> space.
> > > > > > >> > > > > > > * Can take captures of
images/dom elements
> > (including
> > > > > > canvas)
> > > > > > >> > that
> > > > > > >> > > > are
> > > > > > >> > > > > > > lager than the actual
screen size
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > CaptureAndCompare():
> > > > > > >> > > > > > > * All the functionality
of Capture()
> > > > > > >> > > > > > > * Perform a comparison
between the captured image
> > and
> > > a
> > > > > > >> baseline
> > > > > > >> > > > image
> > > > > > >> > > > > > > located at the specified
location in either the
> > assets
> > > > > > folder
> > > > > > >> or
> > > > > > >> > > the
> > > > > > >> > > > > > > sdcard.
> > > > > > >> > > > > > > * User can specify per
color channel tolerances as
> > > well
> > > > as
> > > > > > >> total
> > > > > > >> > > > pixel
> > > > > > >> > > > > > > tolerances to avoid false
positives for the
> > inevitable
> > > > > > >> rendering
> > > > > > >> > > > > > > differences across devices.
> > > > > > >> > > > > > > * Optionally output a
png file that contains the
> > > > > differences
> > > > > > >> > > between
> > > > > > >> > > > > the
> > > > > > >> > > > > > > actual and the baseline
for debugging/triage
> > purposes,
> > > > two
> > > > > > >> modes:
> > > > > > >> > > > > binary
> > > > > > >> > > > > > > diff (all failing pixels
appear as solid white) or
> > the
> > > > > true
> > > > > > >> > > > differences
> > > > > > >> > > > > > > between pixels.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > If you can spare some
time, I would love it if you
> > > could
> > > > > > take
> > > > > > >> a
> > > > > > >> > > look
> > > > > > >> > > > at
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > api and parameters I
have defined to make sure
> they
> > > > adhere
> > > > > > to
> > > > > > >> > > Cordova
> > > > > > >> > > > > > > plugin best practices.
 The most crucial part
> would
> > be
> > > > in
> > > > > > the
> > > > > > >> > > plugin
> > > > > > >> > > > > > itself
> > > > > > >> > > > > > > at ScreenCapture.java:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/Charbs09/Cordova-Mobile-Spec-ScreenCapture/blob/master/src/org/apache/cordova/plugin/ScreenCapture.java(capture()
> > > > > > >> > > > > > > and captureAndCompare()
are the two exposed
> > functions)
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I'm also interested to
know you thoughts on it's
> > > usage.
> > > >  I
> > > > > > >> > started
> > > > > > >> > > > with
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > mobile-spec testing framework
and put some quick
> > > > rendering
> > > > > > >> tests
> > > > > > >> > > into
> > > > > > >> > > > > the
> > > > > > >> > > > > > > Autotest section as a
viable use case.  In order
> to
> > > get
> > > > > the
> > > > > > >> > WebView
> > > > > > >> > > > > into
> > > > > > >> > > > > > > the state I wanted to
capture, I had to implement
> a
> > > wait
> > > > > > >> timer to
> > > > > > >> > > > allow
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > view to update before
taking the capture.  Once
> the
> > > wait
> > > > > > >> function
> > > > > > >> > > > > > > completes, the capture
can be taken and everything
> > > from
> > > > > > there
> > > > > > >> is
> > > > > > >> > > > > callback
> > > > > > >> > > > > > > based.  I use the Jasmine
waitsFor()/runs() blocks
> > to
> > > > make
> > > > > > the
> > > > > > >> > > tests
> > > > > > >> > > > > run
> > > > > > >> > > > > > > synchronously.  For some
validation testing I
> made a
> > > > > compare
> > > > > > >> > > function
> > > > > > >> > > > > in
> > > > > > >> > > > > > > Javascript to run against
the native compare.
>  Turns
> > > out
> > > > > > Java
> > > > > > >> > runs
> > > > > > >> > > > alot
> > > > > > >> > > > > > > faster than Javascript
(surprise!) thus the native
> > > > compare
> > > > > > is
> > > > > > >> > about
> > > > > > >> > > > > 2-5x
> > > > > > >> > > > > > > faster.  All in all the
process is fairly quick,
> the
> > > > full
> > > > > > >> > > > > > captureAndCompare
> > > > > > >> > > > > > > with 3 file io's takes
about 233ms on a Nexus One,
> > and
> > > > > > ~100ms
> > > > > > >> on
> > > > > > >> > a
> > > > > > >> > > > > Nexus
> > > > > > >> > > > > > 10
> > > > > > >> > > > > > > (256x256 image size).
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Anyways here is the usage:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/Charbs09/Cordova-Mobile-Spec-ScreenCapture/blob/master/assets/www/autotest/tests/rendering.tests.js
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > And here's the JS wrapper
for the plugin calls:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/Charbs09/Cordova-Mobile-Spec-ScreenCapture/blob/master/assets/www/screencapture.js
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for your time,
and I look forward to ANY
> > > feedback
> > > > > > >> positive
> > > > > > >> > > or
> > > > > > >> > > > > > > negative.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks,
> > > > > > >> > > > > > > Aaron
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message