Mailing-List: contact reviews-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: reviews@aurora.apache.org
Content-Type: multipart/alternative;
 boundary="===============4386118059195973308=="
MIME-Version: 1.0
Subject: Re: Review Request 59699: Improve task history pruning by batch
 deleting tasks
From: Zameer Manji <zmanji@apache.org>
To: David McLaughlin <david@dmclaughlin.com>, Stephan Erb <serb@apache.org>,
 Zameer Manji <zmanji@apache.org>, Santhosh Kumar <kssanthoshkumar@gmail.com>
Cc: Aurora <reviews@aurora.apache.org>, Kai Huang <texasred2013@hotmail.com>
Date: Fri, 02 Jun 2017 18:54:22 -0000
Message-ID: <20170602185422.16397.66234@reviews-vm2.apache.org>
Auto-Submitted: auto-generated
Sender: Zameer Manji <noreply@reviews.apache.org>
References: <20170602000146.34605.10866@reviews-vm2.apache.org>
In-Reply-To: <20170602000146.34605.10866@reviews-vm2.apache.org>
Reply-To: Zameer Manji <zmanji@apache.org>
archived-at: Fri, 02 Jun 2017 18:54:28 -0000

--===============4386118059195973308==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59699/#review176827
-----------------------------------------------------------


Ship it!


Ship It!

- Zameer Manji


On June 1, 2017, 5:01 p.m., Kai Huang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59699/
> -----------------------------------------------------------
> 
> (Updated June 1, 2017, 5:01 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar, Stephan Erb, and Zameer Manji.
> 
> 
> Bugs: AURORA-1929
>     https://issues.apache.org/jira/browse/AURORA-1929
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Improve task history pruning by batch deleting tasks.
> 
> The `'aurora_admin prune_tasks'` endpoint seems to be very slow when the cluster has a large number of inactive tasks.
> 
> This CR batches all removeTasks operations and execute them all at once to avoid additional cost of coalescing. The fix will also benefit implicit task history pruning since it has similar underlying implementation. See https://issues.apache.org/jira/browse/AURORA-1929 for more information and details.
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 73878210f9028901fda3b08e66c6a63c24260d35 
> 
> 
> Diff: https://reviews.apache.org/r/59699/diff/5/
> 
> 
> Testing
> -------
> 
> __unit_tests:__
> 
> ./build-support/jenkins/build.sh
> 
> No unit tests were created for this patch since it does not add new functionalities or alter the interface, but improves the efficiency of the existing code.
> 
> __e2e tests:__
> 
> Attached was a screenshot of the task history pruning benchmark obtained from a scale test in Twitter's test cluster.
> 
> - Before applying this patch, the task history pruning takes ~30 minutes on 130K tasks.
> 
> - After applying the patch, the pruning takes ~1 minute.
> 
> 
> File Attachments
> ----------------
> 
> task_history_pruning_benchmark.png
>   https://reviews.apache.org/media/uploaded/files/2017/06/01/74eb5104-d338-4530-abd2-b82fbdc6bf84__task_history_pruning_benchmark.png
> 
> 
> Thanks,
> 
> Kai Huang
> 
>


--===============4386118059195973308==--