pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi" <knogu...@apache.org>
Subject Re: Review Request 22535: PIG-3975: Multiple Scalar reference calls leading to missing records
Date Tue, 17 Jun 2014 15:34:11 GMT


> On June 15, 2014, 3:57 p.m., Rohini Palaniswamy wrote:
> > test/org/apache/pig/test/Util.java, line 849
> > <https://reviews.apache.org/r/22535/diff/1/?file=608350#file608350line849>
> >
> >     comp.aggregateScalarsFiles(); here?

I'm not familiar with the code, but it seems like test has two version of MRPlan creation.
buildMRPlan() and buildMRPlanWithOptimizer().   Only the latter contains comp.aggregateScalarsFiles.

Since it was like this BEFORE my patch, I wanted to keep it that way.
I needed to add comp.connectSoftLink() since this was taken out from comp.compile() by my
patch.

Having said that, I still need to run a full unit/e2e test.


>    853     public static MROperPlan buildMRPlan(PhysicalPlan pp, PigContext pc) throws
Exception{
>    854         MRCompiler comp = new MRCompiler(pp, pc);
>    855         comp.compile();
> +  856         comp.connectSoftLink();
>    857         return comp.getMRPlan();
>    858     }
>    859
>    860     public static MROperPlan buildMRPlanWithOptimizer(PhysicalPlan pp, PigContext
pc) throws Exception {
>    861         MapRedUtil.checkLeafIsStore(pp, pc);
>    862
>    863         MapReduceLauncher launcher = new MapReduceLauncher();
>    864
>    865         java.lang.reflect.Method compile = launcher.getClass()
>    866                 .getDeclaredMethod("compile",
>    867                         new Class[] { PhysicalPlan.class, PigContext.class });
>    868
>    869         compile.setAccessible(true);
>    870
>    871         return (MROperPlan) compile.invoke(launcher, new Object[] { pp, pc });
>    872     }


- Koji


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22535/#review45552
-----------------------------------------------------------


On June 12, 2014, 9:51 p.m., Koji Noguchi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22535/
> -----------------------------------------------------------
> 
> (Updated June 12, 2014, 9:51 p.m.)
> 
> 
> Review request for pig, Daniel Dai and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-3975
>     https://issues.apache.org/jira/browse/PIG-3975
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> PIG-3975: Multiple Scalar reference calls leading to missing records
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 51014eb

>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
e6a4261 
>   test/org/apache/pig/test/TestFRJoin2.java c32a2c5 
>   test/org/apache/pig/test/Util.java 97c45c7 
> 
> Diff: https://reviews.apache.org/r/22535/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Koji Noguchi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message