pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park" <piaozhe...@gmail.com>
Subject Re: Review Request 16926: PIG-3557 Implement LIMIT optimizations in Tez
Date Thu, 16 Jan 2014 01:08:02 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16926/#review31980
-----------------------------------------------------------

Ship it!


I will commit it after running tests.

- Cheolsoo Park


On Jan. 16, 2014, 12:44 a.m., Alex Bain wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16926/
> -----------------------------------------------------------
> 
> (Updated Jan. 16, 2014, 12:44 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-3557
>     https://issues.apache.org/jira/browse/PIG-3557
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Implement LIMIT optimizations in Tez - https://issues.apache.org/jira/browse/PIG-3557
> 
> 1. If the previous Tez vertex has a requestedParallelism of 1 and does not start with
a POLoad, we don't need to add a second LIMIT vertex (since the LIMIT we put at the end of
the previous vertex is good enough).
> 
> 2. If we are not in the "limited order by" case, instead of the regular shuffle-sort
edge we can use an unsorted shuffle edge.
> --This code is added but commented out, since it depends on TEZ-661.
> 
> 3. I manually verified that the LimitOptimizer can push LIMIT to the InputHandler in
certain cases (no code changes).
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 5c6a242 
>   test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 9cf5baf 
> 
> Diff: https://reviews.apache.org/r/16926/diff/
> 
> 
> Testing
> -------
> 
> TestTezCompiler unit test updated
> ant test-tez passes
> e2e tests - same results as in clean tez branch
> 
> 
> Thanks,
> 
> Alex Bain
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message