aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David McLaughlin <da...@dmclaughlin.com>
Subject Re: Review Request 56629: Expose task pruning endpoint in aurora_admin
Date Tue, 14 Feb 2017 04:52:50 GMT


> On Feb. 14, 2017, 4:48 a.m., Santhosh Kumar Shanmugham wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
lines 1128-1129
> > <https://reviews.apache.org/r/56629/diff/5/?file=1632842#file1632842line1128>
> >
> >     Depending on th query this can be a full-scan on the store, which can be inefficient.
As mentioned in https://reviews.apache.org/r/56575/ this approach can cause GC pressure. We
should use the `TaskStore.pruneHistory` implementation which might be more efficient.

pruneHistory would search for candidates that are eligible for pruning according to the pruning
settings. This is a way to circumvent the grace periods and other limits, and prune all terminal
tasks that match the provided query. Since this would be called explicitly and can be supplied
with a limit, the operator is able to manage the GC pressure themselves.


- David


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165440
-----------------------------------------------------------


On Feb. 14, 2017, 4:28 a.m., David McLaughlin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> -----------------------------------------------------------
> 
> (Updated Feb. 14, 2017, 4:28 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in order to 'clean
up' after a test run, but also useful in production if you have a bad actor inflating the
size of your task index.
> 
> 
> Diffs
> -----
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1

>   src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java a211483caf59755b1f9b286b58f183e00db6eee6

>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b

>   src/main/python/apache/aurora/client/api/__init__.py 1250ccd16f906eec08c159df3300d1b94b306d8e

>   src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 66abade378d3974dbf031a7bbed9cd6fc4f22d5f

>   src/test/python/apache/aurora/client/api/test_scheduler_client.py fab97986dcecf7761e161f4074d5108a8c362e6e

> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> -------
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works as expected.

> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message