falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 42188: FALCON-1742 Implement instance summary api for native scheduler
Date Tue, 19 Jan 2016 08:26:42 GMT


> On Jan. 18, 2016, 11:01 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java,
line 414
> > <https://reviews.apache.org/r/42188/diff/1/?file=1194070#file1194070line414>
> >
> >     Shouldn't this be outside for loop?
> 
> Pallavi Rao wrote:
>     In this implementation, I'm retrieving instances per state, rather than retrieve
all instances and sort and count in memory.
> 
> Deepak Barr wrote:
>     states variable is different for different iterations. Should be OK.
> 
> Ajay Yadava wrote:
>     The ordering can be done in db query, that way all invocations of the query will
result in sorted result and we will not need to make multiple db calls. That won't work?
> 
> Pallavi Rao wrote:
>     Yeah.. it will sort, but, I'll still have to iterate over the entire result set or
issue a group command. Also, the result set can be large. If it is limited to a single state,
it gets returned faster. It requires additional changes to the StateStore layer. Hence, the
simplistic approach.
> 
> Ajay Yadava wrote:
>     So in this approach we will fire approximately 8 queries(1 corresponding to each
state), we will fetch all the columns, convert them to POJOs whereas we need just count. What
I am suggesting is just fire 1 query which will group by state and return the count corresponding
to each state and this will be several times faster and simpler. Another difference is that
for a large range current approach may cause memory issues(and severe performance degradation)
as it fetches all columns and all rows whereas the suggested approach is immune to such cases.

Yep. Makes sense. It just wasteful and can cause issues. Will address


- Pallavi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42188/#review114982
-----------------------------------------------------------


On Jan. 12, 2016, 10:14 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42188/
> -----------------------------------------------------------
> 
> (Updated Jan. 12, 2016, 10:14 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1742
>     https://issues.apache.org/jira/browse/FALCON-1742
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> This was not implemented.
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/resource/EntitySummaryResult.java 4a885ec 
>   scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java
bffdb0b 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java a82cf03 
>   webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java
b1c8ce0 
> 
> Diff: https://reviews.apache.org/r/42188/diff/
> 
> 
> Testing
> -------
> 
> Added IT and did manual testing.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message