pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy" <rohini.adi...@gmail.com>
Subject Re: Review Request 29644: Port local mode tests to Tez - part4
Date Tue, 20 Jan 2015 23:46:41 GMT


> On Jan. 13, 2015, 10:52 p.m., Rohini Palaniswamy wrote:
> > trunk/test/org/apache/pig/test/TestMultiQueryLocal.java, line 447
> > <https://reviews.apache.org/r/29644/diff/1/?file=808163#file808163line447>
> >
> >     If the location is different why do we have to pass two diffent keys for two
different output formats to get the test passing?
> 
> Daniel Dai wrote:
>     This test is to make sure different storer is using different context. So I use different
key for different storer for the test. There is a difference which makes the test pass in
MR. In tez, OutputFormat and Storer is using the same context, while MR is not. I don't think
this is a big issue.
> 
> Rohini Palaniswamy wrote:
>     Don't we clone the context and use it for different Storers/OutputFormats? May be
that is missing in Tez.  We should fix it as using of same context by different Storers will
cause settings to be overwritten.
> 
> Daniel Dai wrote:
>     Didn't get it. Different Storer/OutputFormat should use independent context. That
is, if you store some config entry in Storer1/OutputFormat1, Storer2/OutputFormat2 will not
see. That is same in both MR and Tez. What's different is, in MR, also provide separation
between Storer1/OutputFormat1. If you store config in OutputFormat1, Storer1 will not see.
But that does not seems to be a required feature.

Oh. That should be fine. Infact HCat folks were asking me for the same thing. They were adding
something to config in RecordWriter and wanted that to be available to commitJob, but was
not possible with MR


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29644/#review67958
-----------------------------------------------------------


On Jan. 20, 2015, 10:08 p.m., Daniel Dai wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29644/
> -----------------------------------------------------------
> 
> (Updated Jan. 20, 2015, 10:08 p.m.)
> 
> 
> Review request for pig and Rohini Palaniswamy.
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> See PIG-4359
> 
> 
> Diffs
> -----
> 
>   trunk/shims/test/hadoop23/org/apache/pig/test/MiniCluster.java 1652696 
>   trunk/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1652696 
>   trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezCompiler.java 1652696

>   trunk/test/excluded-tests-20 1652696 
>   trunk/test/org/apache/pig/test/MiniGenericCluster.java 1652696 
>   trunk/test/org/apache/pig/test/TestMultiQueryLocal.java 1652696 
>   trunk/test/org/apache/pig/test/TestPOPartialAggPlan.java 1652696 
>   trunk/test/org/apache/pig/test/TestPOPartialAggPlanMR.java PRE-CREATION 
>   trunk/test/org/apache/pig/test/TestStore.java 1652696 
>   trunk/test/org/apache/pig/test/TestStoreBase.java PRE-CREATION 
>   trunk/test/org/apache/pig/test/TestStoreLocal.java PRE-CREATION 
>   trunk/test/org/apache/pig/tez/TestPOPartialAggPlanTez.java PRE-CREATION 
>   trunk/test/org/apache/pig/tez/TezUtil.java PRE-CREATION 
>   trunk/test/tez-local-tests 1652696 
> 
> Diff: https://reviews.apache.org/r/29644/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message