pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy" <rohini.adi...@gmail.com>
Subject Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime
Date Tue, 04 Feb 2014 22:08:44 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/
-----------------------------------------------------------

(Updated Feb. 4, 2014, 10:08 p.m.)


Review request for pig, Cheolsoo Park and Daniel Dai.


Changes
-------

Patch that deletes and newly adds util classes in util package instead of svn mv.


Bugs: PIG-3742
    https://issues.apache.org/jira/browse/PIG-3742


Repository: pig


Description
-------

Changes made:
1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex
and Edge.
2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package.
Does not show up cleanly in review board. Will be doing a svn mv while committing.
3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected
in the second edge when the parallelism of first vertex changed after input split calculation.
Also made POIdentityOutTez work with shuffle input as well when trying to test performance
with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner
or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for
L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping
unsorted shuffle with TEZ-661 might make it better.


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java
1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java
PRE-CREATION 

Diff: https://reviews.apache.org/r/17681/diff/


Testing
-------

Unit and tez.conf e2e tests pass.


Thanks,

Rohini Palaniswamy


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message