openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sven Lange-Last" <>
Subject Re: A direction to take in a new component, "the scheduler"
Date Thu, 18 Jul 2019 17:49:16 GMT
Hello Dominic and other discussion participants,

we are in the great situation that Dominic (style95) volunteers to create 
and verify an alternative OpenWhisk architecture that may solve some of 
the problems that adopters of OpenWhisk may observe when running 
production systems with high utilization. I really appreciate that Dominic 
already invested and is willing to invest so much time.

While I encourage Dominic to experiment with a different architecture, my 
point of view is that it's in the interest of the project as well as all 
adopters running OpenWhisk-based production systems to keep the project 
stable. We need an approach where we can continue to evolve the existing 
architecture and provide bug fixes without being slowed down or affected 
by the work on or problems with the new architecture. At the same time, we 
need a playground where we can work on the new architecture effectively 
without negatively affecting the existing architecture.

For me, this means:

* We need feature flags / switches as suggested by Dominic and others that 
keep the existing architecture enabled by default.

* The standard deployment only configures and installs what is needed for 
the existing architecture. Installing extra components like etcd takes 
more time so that development, build and test cycles take longer. In 
addition, extra components consume more resources so that brittle system 
tests for the existing architecture may fail more often.

* We need a good separation between tests for existing and new 
architecture. System tests are often brittle and need special resources 
like etcd. Such tests for the new architecture must not be part of the 
existing gradle suites. I think that stable, fast-running unit tests can 
be part of existing suites.

* The Travis checks for pull requests must not contain system tests or 
other brittle / long-running tests of the new architecture. We already 
have some brittle tests today that occasionally fail Travis. As Dominic 
suggested, extra build / deployment / test jobs that run separately would 
be a great idea.

I would like to add a different aspect...

My experience is that the observed problems of an OpenWhisk based system 
really depend on utilization and the workload mix (memory limits, action 
duration, action container image, number of distinct actions, ...). And I 
would expect that people running a few activations will have totally 
different observations than those people running a system with hundreds of 
invokers and millions of activations with a diverse workload mix. And 
based on the observed problems, different contributors will probably try 
to solve different problems and set different priorities.

I had a great discussion with Dominic on these aspects. It turned out that 
he observes different problems than I do when running a large production 
environment. I'm very interested in Dominic's proposal - at the same time, 
my impression is that it will not necessarily address the problems I see. 
Still, we can learn a lot from Dominic's work.

At some point in time, the OpenWhisk community probably needs to decide 
whether to keep the existing architecture and evolve it - or to replace it 
with the suggested architecture if it proves useful. My impression is that 
Dominic only received limited feedback on his proposal and I'm guilty of 
providing my feedback much too late. We should discuss the problems we 
observe with running OpenWhisk more openly so that contributors can align 
their work with the need of others.

Mit freundlichen Grüßen / Regards,

Sven Lange-Last
Senior Software Engineer
IBM Cloud Functions
Apache OpenWhisk

Find me on:  

Schoenaicher Str. 220
Boeblingen, 71032

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, 
HRB 243294

View raw message