aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Cohen <>
Subject Re: aurora job scalability
Date Fri, 18 Mar 2016 16:27:52 GMT
Hi Christopher,

I think you already got an answer from Stephan in IRC, but just wanted to
follow up for the sake of posterity (in case anyone in the future has a
similar question and finds this thread). The only limit on the number of
jobs that Aurora can run would currently be the amount of memory available
to the Scheduler. Suffice it to say that at Twitter we're running thousands
of jobs with no issues.

Let us know if you have any follow up questions.



On Fri, Mar 18, 2016 at 9:12 AM, Christopher M Luciano <
> wrote:

>  Hi all. It seems that we may be outgrowing Marathon. We have a problem
> with the amount of application that we are using, causing us to not exactly
> be "compliant" with Marathon goals. It seems that the unit for Aurora is a
> job+instance of that job. Does job map to a Marathon application? If
> similar is there a known limitation to how many jobs one can have?
>  What we discovered for Marathon more application+bigger env_vars = bigger
> zknode size and we come dangerously close to hitting the 1 MB default of
> the zk zknode size. I'm wondering if this type of a thing has potentially
> been fixed already in Aurora.
> Christopher M Luciano
> Staff Software Engineer, Platform Services
> IBM Watson Core Technology

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message