aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Erb <s...@apache.org>
Subject Re: Review Request 59640: Prioritize adding instances over updating instances during an update
Date Wed, 31 May 2017 22:28:19 GMT


> On May 31, 2017, 12:13 a.m., Stephan Erb wrote:
> > Thanks for the patch! That is an interesting idea. I am wondering if it is safe
in all cases. For example, if I reduce the resource requirements while increasing the number
of instances, my service could potentially breach his assigned quota. 
> > 
> > The "sliding update" Bill talks about in https://www.youtube.com/watch?v=y1hi7K1lPkk&t=13m25s
might be a more robust alternative. It will be more complicated to implement though...
> > 
> > I am not working on very large clusters, so I am kind of interested what the others
have to say here.
> 
> Jordan Ly wrote:
>     Thanks for the comment Stephan! You bring up a good point.
>     
>     From what I understand, your final configuration cannot exceed your production quota
no matter what. If you reduce the number of resource requirements while increasing the number
of instances, if the overall end resource usage is over the quota your update will be denied.
>     
>     In the scenario where your end state usage is below your production usage, you may
temporarily breach your quota as the update rolls out. For example, if you add new instances
and change the resource requirements of all instances, the new instances will be added before
the old ones can update. However, at the end of the update, you will be within your quota.

>     
>     This scenario is already occuring with the current ordering (but with killing instances).
Right now, we update in a numerical order meaning killing instances happens last. Let's say
you increase the resource requirements but reduce the number of instances: at some point during
the update you will temporarily exceed your quota as you have updated old instance with more
resources (pushing total utilization over the quota) but have yet to kill other instances
(bringing utilization back down to end state within quota).
> 
> David McLaughlin wrote:
>     This is correct. Test coverage for this is here: https://github.com/apache/aurora/blob/master/src/test/java/org/apache/aurora/scheduler/quota/QuotaManagerImplTest.java#L593

Oh right. Thanks for the clarification!


- Stephan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59640/#review176371
-----------------------------------------------------------


On May 30, 2017, 11:21 p.m., Jordan Ly wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59640/
> -----------------------------------------------------------
> 
> (Updated May 30, 2017, 11:21 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, Stephan Erb,
and Zameer Manji.
> 
> 
> Bugs: AURORA-1928
>     https://issues.apache.org/jira/browse/AURORA-1928
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Currently, when updating a job we choose to update instances naively by ascending instance
ID number.
> However, it would be better to add new instances before killing and updating older instances.
> 
> This patch makes it so the job updater prefers to create new instances, then
> update instances, and finally kill instances.
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 14c2d2de3186271819a5f4e527d3b30fd34d2b21

>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 290385d737294e23e9dd50f2631303124aa7af7c

>   src/test/java/org/apache/aurora/scheduler/updater/UpdateFactoryImplTest.java 611f6b8681c8e0b286cd361bdb5ace1fea39d9a5

> 
> 
> Diff: https://reviews.apache.org/r/59640/diff/1/
> 
> 
> Testing
> -------
> 
> Ran unit tests and integration tests.
> 
> Had to modify some integration tests since we now prefer to create over update -- needed
to change
> the ordering of actions. Additionally, some unit tests only specified configs for one
instance even
> though desiredInstances is always 2 -- had to make it so the range of configurations
is always 0-2
> when creating. Otherwise, it would try to create instances first even though the test
didn't really
> care.
> 
> Tested different update configurations on the Vagrant cluster: only adding instances,
only updating
> instances, only killing instances, create & update, update & kill.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message