hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Chauhan <hashut...@apache.org>
Subject Re: Review Request 56810: Compute table stats when user computes column stats
Date Fri, 24 Feb 2017 23:29:32 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/#review166776
-----------------------------------------------------------




ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java (line 72)
<https://reviews.apache.org/r/56810/#comment238861>

    Why is that. Lets say you have 
    Table t1 has 2 partitions p1=1 and p1=2 and user runs
    analyze table t1 partition (p1=2) for columns
    
    In this case you can still collect basic stats (numrows etc.) for p1=2. Basic stats wont
be collected for p1=1 but that should be fine.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10250)
<https://reviews.apache.org/r/56810/#comment238855>

    Won't analyzeRewrite always be null when isAnyalzeCommand() is false. If so, this change
is not needed.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10257)
<https://reviews.apache.org/r/56810/#comment238858>

    tblSpec cant be null at this point. It will already be constructed with all the info needed
during getMetaData(). Can remove this reconstruction of tblSpec.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10262)
<https://reviews.apache.org/r/56810/#comment238859>

    LOG.debug()



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10320)
<https://reviews.apache.org/r/56810/#comment238860>

    LOG.debug ("Adding partition: {}", partn);



ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java (lines 307 - 343)
<https://reviews.apache.org/r/56810/#comment238862>

    This logic of generating tasks should be migrated to TezCompiler::ProcessAnalyzeTable()

    Currently its breaking the abstraction.


- Ashutosh Chauhan


On Feb. 23, 2017, 6:52 p.m., pengcheng xiong wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56810/
> -----------------------------------------------------------
> 
> (Updated Feb. 23, 2017, 6:52 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-15903
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 93b8183

>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2430811 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5f9ccc8 
>   ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/exec_parallel_column_stats.q ceacc24 
>   ql/src/test/results/clientpositive/alter_table_update_status.q.out a3c4f1a 
>   ql/src/test/results/clientpositive/analyze_tbl_date.q.out a0cdbca 
>   ql/src/test/results/clientpositive/analyze_tbl_part.q.out ed90b6f 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 32644dc 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out a8e4854 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 31c4ed1 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out 5d4fe6c 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out b4d46d2 
>   ql/src/test/results/clientpositive/annotate_stats_limit.q.out ea181cb 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1ab 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f 
>   ql/src/test/results/clientpositive/annotate_stats_union.q.out 059f261 
>   ql/src/test/results/clientpositive/autoColumnStats_3.q.out dca158b 
>   ql/src/test/results/clientpositive/avro_decimal.q.out 64e65ca 
>   ql/src/test/results/clientpositive/avro_decimal_native.q.out cebc342 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out f260f03 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
>   ql/src/test/results/clientpositive/colstats_all_nulls.q.out f67f81b 
>   ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out af21343

>   ql/src/test/results/clientpositive/column_table_stats.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out 5c9f901 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 08fb42e 
>   ql/src/test/results/clientpositive/columnstats_quoting.q.out 52e3538 
>   ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 60652e0 
>   ql/src/test/results/clientpositive/compustat_avro.q.out 50c03bd 
>   ql/src/test/results/clientpositive/compute_stats_date.q.out 73f08ad 
>   ql/src/test/results/clientpositive/constGby.q.out fd8ecc2 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out ecf6e57 
>   ql/src/test/results/clientpositive/decimal_stats.q.out cb02f76 
>   ql/src/test/results/clientpositive/deleteAnalyze.q.out 4382522 
>   ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out e10edb4 
>   ql/src/test/results/clientpositive/distinct_stats.q.out 73b4add 
>   ql/src/test/results/clientpositive/drop_table_with_stats.q.out 52aa10a 
>   ql/src/test/results/clientpositive/exec_parallel_column_stats.q.out f256ec1 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out b12d3a1 
>   ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out 85d7dc4

>   ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
80ccddd 
>   ql/src/test/results/clientpositive/llap/columnstats_part_coltype.q.out 3e28e58 
>   ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 
>   ql/src/test/results/clientpositive/llap/drop_partition_with_stats.q.out c6ab40d 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 67f960a 
>   ql/src/test/results/clientpositive/llap/llap_stats.q.out f6921f1 
>   ql/src/test/results/clientpositive/llap/llapdecider.q.out d514f42 
>   ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out c8190bd 
>   ql/src/test/results/clientpositive/llap/schema_evol_stats.q.out be8ca4a 
>   ql/src/test/results/clientpositive/llap/special_character_in_tabnames_1.q.out d01b373

>   ql/src/test/results/clientpositive/llap/stats_only_null.q.out c905ceb 
>   ql/src/test/results/clientpositive/llap/union_remove_26.q.out 18b9aa5 
>   ql/src/test/results/clientpositive/llap/vector_outer_join1.q.out 2ea8c8f 
>   ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out 2077dce 
>   ql/src/test/results/clientpositive/llap/vector_outer_join3.q.out dbbfd34 
>   ql/src/test/results/clientpositive/llap/vector_outer_join4.q.out ffce9e6 
>   ql/src/test/results/clientpositive/llap/vector_outer_join5.q.out 4f25253 
>   ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
c9eec63 
>   ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction2.q.out
062fef6 
>   ql/src/test/results/clientpositive/metadata_only_queries.q.out 57b59dd 
>   ql/src/test/results/clientpositive/partial_column_stats.q.out 59b52b0 
>   ql/src/test/results/clientpositive/pcs.q.out c422a3d 
>   ql/src/test/results/clientpositive/remove_exprs_stats.q.out 8fe688d 
>   ql/src/test/results/clientpositive/rename_table_update_column_stats.q.out c025cfa 
>   ql/src/test/results/clientpositive/spark/annotate_stats_join.q.out 31ccc5c 
>   ql/src/test/results/clientpositive/spark/avro_decimal_native.q.out cebc342 
>   ql/src/test/results/clientpositive/spark/limit_pushdown.q.out c81240d 
>   ql/src/test/results/clientpositive/spark/metadata_only_queries.q.out 543d0ef 
>   ql/src/test/results/clientpositive/spark/stats_only_null.q.out 359eea3 
>   ql/src/test/results/clientpositive/spark/vector_elt.q.out b49462a 
>   ql/src/test/results/clientpositive/spark/vector_left_outer_join.q.out 91af229 
>   ql/src/test/results/clientpositive/spark/vector_outer_join1.q.out ca4ce15 
>   ql/src/test/results/clientpositive/spark/vector_outer_join2.q.out 8a9f90f 
>   ql/src/test/results/clientpositive/spark/vector_outer_join3.q.out dbbfd34 
>   ql/src/test/results/clientpositive/spark/vector_outer_join4.q.out ffce9e6 
>   ql/src/test/results/clientpositive/spark/vector_outer_join5.q.out 4f25253 
>   ql/src/test/results/clientpositive/spark/vectorization_0.q.out 5086f53 
>   ql/src/test/results/clientpositive/spark/vectorization_13.q.out a6ef031 
>   ql/src/test/results/clientpositive/spark/vectorization_14.q.out 1541908 
>   ql/src/test/results/clientpositive/spark/vectorization_15.q.out 1d925c5 
>   ql/src/test/results/clientpositive/spark/vectorization_16.q.out e731c2d 
>   ql/src/test/results/clientpositive/spark/vectorization_17.q.out a8f401b 
>   ql/src/test/results/clientpositive/spark/vectorization_9.q.out e731c2d 
>   ql/src/test/results/clientpositive/spark/vectorization_div0.q.out baea88f 
>   ql/src/test/results/clientpositive/spark/vectorization_pushdown.q.out 1f1bb30 
>   ql/src/test/results/clientpositive/spark/vectorization_short_regress.q.out 47664ce

>   ql/src/test/results/clientpositive/spark/vectorized_case.q.out 67ed3dd 
>   ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out 030a71b 
>   ql/src/test/results/clientpositive/spark/vectorized_math_funcs.q.out 90ef576 
>   ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 996021f 
>   ql/src/test/results/clientpositive/spark/vectorized_shufflejoin.q.out 5930057 
>   ql/src/test/results/clientpositive/spark/vectorized_string_funcs.q.out 26aab1c 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_2.q.out 8cb04c7 
>   ql/src/test/results/clientpositive/stats_invalidation.q.out a0e7663 
>   ql/src/test/results/clientpositive/stats_missing_warning.q.out 0ed70a0 
>   ql/src/test/results/clientpositive/stats_only_null.q.out 88c2114 
>   ql/src/test/results/clientpositive/stats_partial_size.q.out c779741 
>   ql/src/test/results/clientpositive/stats_ppr_all.q.out d4060c6 
>   ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out c7ee93d

>   ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out dd8849d 
>   ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
>   ql/src/test/results/clientpositive/tez/explainuser_3.q.out ef71d73 
>   ql/src/test/results/clientpositive/vector_outer_join1.q.out 3df63dc 
>   ql/src/test/results/clientpositive/vector_outer_join2.q.out ec7de23 
>   ql/src/test/results/clientpositive/vector_outer_join3.q.out 1d2abee 
>   ql/src/test/results/clientpositive/vector_outer_join4.q.out a73a5e1 
> 
> Diff: https://reviews.apache.org/r/56810/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message