openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Thömmes <markusthoem...@apache.org>
Subject Re: Maximum number of file descriptors available when running actions with managed runtimes
Date Wed, 10 Jul 2019 14:25:52 GMT
Thanks as always for your very detailed writeup and analysis of the issue
at hand! Great job Sven!

It seems to me that we might want to look at analyzing our runtimes for the
number of file-descriptors they open and make up for that for the user. The
user should always get at least the amount of descriptors specified. I
could see us going two ways about this:

1. Determine the number of descriptors per runtime, inform the system about
them (via the manifest) and add these to the limits.
2. Configure the system with enough slack to cover all potential cases.

I can see 1. being quite brittle, given that by looking at your analysis,
it seems like even the host might have an impact here. I'm leaning towards
the 2. option. We should analyze all our runtimes and get a feeling for how
many file descriptors are needed by each. If they are all in the same
ballpark, I'd say let's add ~50fds on top of the limit and document it as
"at least X" vs "exactly X".

Cheers,
Markus

Am Mi., 10. Juli 2019 um 16:20 Uhr schrieb Sven Lange-Last <
sven.lange-last@de.ibm.com>:

> Apache OpenWhisk limits the number of file descriptors available when
> running actions - at least, when using the Docker container factory [1].
> The soft / hard limit for the number of file descriptors that can be open
> for any process in an action container is set to 1024. This value is
> hard-coded. Since there is a system-wide limit for open file descriptors
> on a Linux system, this setting is meant to prevent a single action
> container from opening so many file descriptors that other action
> containers or the whole system are affected.
>
> Apache OpenWhisk documents this limitation in the "OpenWhisk system
> details" document [2]. End users will probably understand this document as
> a kind of promise that up to 1024 file descriptors can be opened by action
> code...
>
> While we currently have a hard-coded limit for open file descriptors in
> action containers, managed action runtime processes consume file
> descriptors to do their work. We have tests that verify a minimum /
> maximum amount of file descriptors can be opened by a default Node.js
> action [3]. These tests already consider that action code won't have the
> full limit of 1024 file descriptors available because the managed runtime
> also opens files [4].
>
> When changing the default Node.js version from 6 to 10, we noticed that
> the Node.js 10 managed runtime consumes 20 instead of 15 and adapted the
> tests accordingly [5]. While decreasing the number of available
> descriptors for action code from 1009 to 1004 is likely no problem, we
> have to be aware that decreasing that number can potentially break
> existing actions and must be considered a regression.
>
> So one purpose of this post is to make managed runtime developers aware
> that changes in the managed runtime process - like the number of file
> descriptors opened by the action process - can break existing actions.
> This also applies to the physical memory usage of the managed runtime
> process.
>
> The other purpose is to start a discussion / hear your opinion on the
> general problem that the file descriptor limit is hard coded at the moment
> and this limit will never be available for action developers because the
> managed runtime already consumes file descriptors.
>
> In addition, I wanted to inform you about a PR I opened today [6]. This PR
> further relaxes the test verifying that a minimum number of file
> descriptors is available - sorry for that. We have test systems where the
> limit tests are failing permanently. Long story in short words: Docker
> 18.09.3 seems to set up action containers differently than earlier
> versions leading to a slightly higher consumption of file descriptors. See
> the PR for all details.
>
>
> [1]
>
> https://github.com/apache/incubator-openwhisk/blob/b0d48c87816f2e7c04ad9b9f9b4844a6e064047f/core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InvokerReactive.scala#L105
> [2]
>
> https://github.com/apache/incubator-openwhisk/blob/ccac7d5131a80bddeb1b6dc0c6580051c90bc264/docs/reference.md
> [3]
>
> https://github.com/apache/incubator-openwhisk/blob/b0d48c87816f2e7c04ad9b9f9b4844a6e064047f/tests/src/test/scala/org/apache/openwhisk/core/limits/ActionLimitsTests.scala#L366-L416
> [4]
>
> https://github.com/apache/incubator-openwhisk/blob/b0d48c87816f2e7c04ad9b9f9b4844a6e064047f/tests/src/test/scala/org/apache/openwhisk/core/limits/ActionLimitsTests.scala#L68-L69
> [5]
>
> https://github.com/apache/incubator-openwhisk/pull/4450#discussion_r277969441
> [6] https://github.com/apache/incubator-openwhisk/pull/4553
>
>
>
> Mit freundlichen Grüßen / Regards,
>
> Sven Lange-Last
> Senior Software Engineer
> IBM Cloud Functions
> Apache OpenWhisk
>
>
> E-mail: sven.lange-last@de.ibm.com
> Find me on:
>
>
> Schoenaicher Str. 220
> Boeblingen, 71032
> Germany
>
>
>
>
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
> HRB 243294
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message