openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michele Sciabarra <mich...@sciabarra.com>
Subject Re: Performance Comparison of Runtimes and Improvements on the ActionLoop Runtime
Date Sun, 25 Nov 2018 21:51:42 GMT
Updates on the performances of the GoLang runtime after a weekend of hacking .

The baseline is the Nodejs runtime:

wrk -t1 -c1 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.86ms  703.95us  18.73ms   98.62%
    Req/Sec     1.22k   154.95     1.42k    82.00%
  12177 requests in 10.00s, 2.65MB read
Requests/sec:   1217.30
Transfer/sec:    271.04KB

First, I cleaned the runtime removing Go Routines, and I reached those numbers:

wrk -t1 -c1 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   688.79us  173.93us   6.29ms   95.28%
    Req/Sec     1.43k   217.80     1.68k    91.30%
  6544 requests in 10.06s, 0.91MB read
Requests/sec:    650.42
Transfer/sec:     92.74KB

A bit better but still not  entirely satisfactory. So I decided to try the plan A (the original
plan!) 
generate an action embedding the http server in the action. Luckily the infrastructure developed
so far made it easy.
I also decided to follow Dragos suggestion using the fasthttp server. And this is the result!

wrk -t1 -c1 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   691.45us    1.28ms  37.10ms   99.42%
    Req/Sec     1.61k   133.34     1.93k    80.20%
  16132 requests in 10.10s, 2.66MB read
Requests/sec:   1597.22
Transfer/sec:    269.84KB

Yup, faster than nodejs. Also, this has the advantage that supports concurrency out of the
box.


So I tried the performances with 10 concurrent requests, and compared with NodeJS with same
level of concurrency:


NodeJS:
wrk -t10 -c10 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.78ms    1.71ms  27.45ms   77.25%
    Req/Sec   269.66     83.11   555.00     70.20%
  26875 requests in 10.01s, 5.84MB read
Requests/sec:   2684.94
Transfer/sec:    597.82KB

Golang Optimized:
wrk -t10 -c10 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.96ms    0.96ms  11.83ms   79.67%
    Req/Sec   523.91     47.02   660.00     66.44%
  52702 requests in 10.11s, 8.70MB read
Requests/sec:   5214.65
Transfer/sec:      0.86MB


5214 requests/second GoLang versus 2684.96 Node JS!


-- 
  Michele Sciabarra
  michele@sciabarra.com

----- Original message -----
From: Michele Sciabarra <michele@sciabarra.com>
To: dev@openwhisk.apache.org
Subject: Performance Comparison of Runtimes  and Improvements on the ActionLoop Runtime
Date: Sat, 24 Nov 2018 15:25:50 +0100

Hello all, 

last week I discovered that, while the Python runtime is pretty slow and using an ActionLoop
runtime improves Python a lot, the NodeJS runtime is damn fast, and the ActionLoop/GoLang
runtime is way behind it.

So I tried to understand what is going on and why there is so much difference between ActionLoop
and NodeJS
Most notably I decided to try to get rid of GoRoutines and just to launch a subprocess from
an HTTP server  and feed data to it.
Hence I wrote a quick-and-dirty prototype to see how fast I can go.

Those are the numbers, comparing raw performances of runtimes.

I measured performances in a similar environment. I run all the tests in Docker for Mac on
a recent MacBook Pro,
using an "hello" action using the command 'wrk -t1 -c1 http://locahost:8080/run' that run
a single thread single connection test for 10 seconds. 

*** nodejs6action
Requests/sec:   1181.36
Transfer/sec:    264.19KB
*** python3action
Requests/sec:     19.80
Transfer/sec:      2.58KB
*** actionloop-golang-v1.11
Requests/sec:     83.80
Transfer/sec:     11.95KB
*** PROTO new actionloop-golang-v1.11
Requests/sec:   1031.38
Transfer/sec:    175.25KB
and this is the current


So while NodeJS will stay faster at 1181 req/sec (well, for a number of reasons), with ActionLoop
(that can work with every language) we can get close at 1031 req/sec


Now I have to work in the runtime to implement the changes I made to the prototype

-- 
  Michele Sciabarra
  michele@sciabarra.com

Mime
View raw message