Mailing-List: contact reviews-help@aurora.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: reviews@aurora.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============4096793130746087338=="
MIME-Version: 1.0
Subject: Re: Review Request 17303: Updated getJobs API to return task stats
 and
 latest task config
From: "Suman Karumuri" <mansu@apache.org>
To: "Bill Farner" <wfarner@apache.org>, "Kevin Sweeney" <kevints@apache.org>
Cc: "Aurora" <reviews@aurora.incubator.apache.org>,
 "Suman Karumuri" <mansu@apache.org>
Date: Fri, 24 Jan 2014 23:19:08 -0000
Message-ID: <20140124231908.19581.97680@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Suman Karumuri" <noreply@reviews.apache.org>
References: <20140124164407.3932.14395@reviews.apache.org>
In-Reply-To: <20140124164407.3932.14395@reviews.apache.org>
Reply-To: "Suman Karumuri" <mansu@apache.org>

--===============4096793130746087338==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit


> On Jan. 24, 2014, 4:44 p.m., Bill Farner wrote:
> > src/main/thrift/org/apache/aurora/gen/api.thrift, line 187
> > <https://reviews.apache.org/r/17303/diff/1/?file=447779#file447779line187>
> >
> >     I think pre-computing these stats restricts the freedom of an API consumer from displaying them differently.  Also, ambiguous names like 'recently' should absolutely be avoided.
> >     
> >     Before we go down this road it would be good to collect real data showing that this aggregation is necessary on the server.  I don't doubt it is, but the exercise will give us a baseline to work with.  If it does prove necessary, it would be good to know what contributes to the problem (data transmission, memory, computation time, etc).
> >     
> >     My intuition is to err on the side of returning more information unless it causes problems; allowing the client to make up its own aggregations.

Sent out detailed stats in an email. If we send all the task configs(6000) instead of summarizing them, the response size for our largest role would be roughly 900kb to 1.2MB through a conservative estimate. Summarizing all these stats on the server, the response size for the current data is 50-100 bytes. On the server side that is not a lot of compute power, but with several concurrent requests, it may saturate the network. On the client side, that is a lot of data and given that JS is single threaded, it will certainly cause UI hiccups and browser crashes. In this particular case, I feel sending summary data is better, since the users have the ability to query the tasks per job using another API and since most of the data we send is discarded in the current use case. 

I agree that recently is ambiguous. We can consider renaming it to tasksFailedInLast6Hours or dropping it entirely. I am in favor of the later since: 
a) We define the status of a service rather arbitrarily. For example, if I have a failed task, that I fixed in a subsequent deployment, the jobs status is still warning for the next 6 hours which is not accurate.
b) I am unsure how useful Stable/UnStable and Warning badges are in practice since most services rely on a stats gathering framework to check the health of their service.


> On Jan. 24, 2014, 4:44 p.m., Bill Farner wrote:
> > src/main/thrift/org/apache/aurora/gen/api.thrift, line 214
> > <https://reviews.apache.org/r/17303/diff/1/?file=447779#file447779line214>
> >
> >     This is overloading the purpose of JobConfiguration.  It's currently "the configuration of a job", and i would prefer to not add use cases to that.
> >     
> >     It would be much tidier to compose this information _with_ JobConfiguration objects in another struct.

Agreed.


- Suman


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17303/#review32720
-----------------------------------------------------------


On Jan. 24, 2014, 7:13 a.m., Suman Karumuri wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17303/
> -----------------------------------------------------------
> 
> (Updated Jan. 24, 2014, 7:13 a.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Bill Farner.
> 
> 
> Bugs: AURORA-64
>     https://issues.apache.org/jira/browse/AURORA-64
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Added task stats to getJobs API so it can be used by the role page in the UI.
> Refactored code from SchedulerzRole and SchedulerzRoleTest into relevant classes so it can be used by the UI and the thrift API.
> Added tests for new code.
> Moved populateJobConfig call into ReadOnlyScheduler.
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/scheduler/base/Jobs.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 06a19d80483b6949c9851b5d38fe34ac712aa75e 
>   src/main/java/org/apache/aurora/scheduler/http/SchedulerzRole.java b0caca73b46fba928fb718ab45a608dad4685a2f 
>   src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java cf9099f307efa23ca34634e3512d9cdbebfa82f2 
>   src/main/thrift/org/apache/aurora/gen/api.thrift 74010379baa2e47cefc228943f766c7b3a8b0d97 
>   src/test/java/org/apache/aurora/scheduler/base/JobsTest.java PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/base/TaskUtil.java PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/http/SchedulerzRoleTest.java 912be189583419e7201e45650d18cd24a6a5a35b 
>   src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 6cefdfad469a9b69a5291ad46be1df14b443472e 
>   src/test/resources/org/apache/aurora/gen/api.thrift.md5 42fdca2759f15d007bee058485c237268c57597a 
> 
> Diff: https://reviews.apache.org/r/17303/diff/
> 
> 
> Testing
> -------
> 
> gradle clean build
> gradle run to test local UI.
> 
> 
> Thanks,
> 
> Suman Karumuri
> 
>


--===============4096793130746087338==--