hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikram Dixit Kumaraswamy" <vikram.di...@gmail.com>
Subject Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Date Wed, 08 Jan 2014 18:51:54 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/#review31347
-----------------------------------------------------------



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
<https://reviews.apache.org/r/16440/#comment59818>

    Need to update java doc.



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
<https://reviews.apache.org/r/16440/#comment59817>

    No param conf.



trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
<https://reviews.apache.org/r/16440/#comment59831>

    spelling error "commone"



trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java
<https://reviews.apache.org/r/16440/#comment59832>

    nit: reword the sentence.



trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
<https://reviews.apache.org/r/16440/#comment59834>

    Could rename the mrTask to be more generic.



trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
<https://reviews.apache.org/r/16440/#comment59836>

    Why doesn't the same apply for the value table desc below?



trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
<https://reviews.apache.org/r/16440/#comment59835>

    Comment needs to be fixed.



trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
<https://reviews.apache.org/r/16440/#comment59838>

    Could add some documentation explaining the use of this class.



trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
<https://reviews.apache.org/r/16440/#comment59839>

    Could it ever be a non-tez task?


- Vikram Dixit Kumaraswamy


On Jan. 7, 2014, 12:50 a.m., Gunther Hagleitner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16440/
> -----------------------------------------------------------
> 
> (Updated Jan. 7, 2014, 12:50 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive
> 
> 
> Description
> -------
> 
> I think the Tez branch is at a point where we can consider merging it back into trunk
after review.
> Tez itself has had its first release, most hive features are available on Tez and the
test coverage is decent. There are a few known limitations, all of which can be handled in
trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require
a branch.)
> Limitations:
> Union all is not yet supported on Tez
> SMB is not yet supported on Tez
> Bucketed map-join is executed as broadcast join (bucketing is ignored)
> Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run
these on MR.
> I am hoping to follow the approach that was taken with vectorization and shoot for a
merge instead of single commit. This would retain history of the branch. Also in vectorization
we required at least three +1s before merge, I'm hoping to go with that as well.
> I will add a combined patch to this ticket for review purposes (not for commit). I'll
also attach instructions to run on a cluster if anyone wants to try.
> 
> 
> Diffs
> -----
> 
>   trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556067 
>   trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556067 
>   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556067 
>   trunk/conf/hive-default.xml.template 1556067 
>   trunk/data/conf/tez/hive-site.xml UNKNOWN 
>   trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556067 
>   trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556067 
>   trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java
1556067 
>   trunk/itests/qtest/pom.xml 1556067 
>   trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556067

>   trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556067 
>   trunk/pom.xml 1556067 
>   trunk/ql/pom.xml 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556067

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556067

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556067

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556067

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION

>   trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556067 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556067

>   trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION

>   trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION

>   trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION 
>   trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556067 
>   trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION 
>   trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION 
>   trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION

>   trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
>   trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION

>   trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION

>   trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION

>   trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION

>   trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION

>   trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION

>   trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out
UNKNOWN 
>   trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION 
>   trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION 
>   trunk/ql/src/test/templates/TestCliDriver.vm 1556067 
>   trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556067 
>   trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556067 
>   trunk/ql/src/test/templates/TestParse.vm 1556067 
>   trunk/ql/src/test/templates/TestParseNegative.vm 1556067 
>   trunk/shims/0.23/pom.xml 1556067 
>   trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556067

> 
> Diff: https://reviews.apache.org/r/16440/diff/
> 
> 
> Testing
> -------
> 
> Testing has been done as a combination of unit tests and q file tests. Unit tests have
been created for new classes where possible, new q file tests cover the delta. I've also gone
through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
> 
> 
> Thanks,
> 
> Gunther Hagleitner
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message