hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongzhi Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8955) alter partition should check for "hive.stats.autogather" in hiveConf
Date Tue, 02 Jun 2015 16:45:49 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569394#comment-14569394
] 

Yongzhi Chen commented on HIVE-8955:
------------------------------------

[~pankit], I think it is by design. The code in HiveAlterHandler.java uses requireCalStats
to check if need to  MetaStoreUtils.updatePartitionStatsFast. 
The requireCalStats will return true if STATS_GENERATED_VIA_STATS_TASK

{noformat}   if(newPart.getParameters().containsKey(StatsSetupConst.STATS_GENERATED_VIA_STATS_TASK))
{
      return true;
    }

{noformat}

The insert overwrite statement will use STATS_GENERATED_VIA_STATS_TASK, hive only create statstask
when HIVESTATSAUTOGATHER is true. As shown in GenMapRedUtils.isMergeRequired code:
{noformat}
 ....     
if (mvTask != null && isInsertTable && hconf.getBoolVar(ConfVars.HIVESTATSAUTOGATHER))
{
        GenMapRedUtils.addStatsTask(fsOp, mvTask, currTask, hconf);
      }
...
{noformat}

Even updatePartitionStatsFast is called in HiveAlterHandler.java, there is no StatsTask when
HIVESTATSAUTOGATHER, the stats are not updated.
Test result:
{noformat}
0: jdbc:hive2://localhost:10000> set hive.stats.autogather= true;
set hive.stats.autogather= true;
No rows affected (0.003 seconds)
0: jdbc:hive2://localhost:10000> insert overwrite table test_part2 partition (x) select
description , code from jspsrcsmall;
insert overwrite table test_part2 partition (x) 
 select description , code from jspsrcsmall;
INFO  : Number of reduce tasks is set to 0 since there's no reduce operator
INFO  : Job running in-process (local Hadoop)
INFO  : 2015-06-02 11:27:22,944 Stage-1 map = 0%,  reduce = 0%
INFO  : Ended Job = job_local688274287_0009
INFO  : Stage-4 is selected by condition resolver.
INFO  : Stage-3 is filtered out by condition resolver.
INFO  : Stage-5 is filtered out by condition resolver.
INFO  : Moving data to: file:/user/hive/warehouse/test_part2/.hive-staging_hive_2015-06-02_11-27-21_770_3880518525165134918-1/-ext-10000
from file:/user/hive/warehouse/test_part2/.hive-staging_hive_2015-06-02_11-27-21_770_3880518525165134918-1/-ext-10002
INFO  : Loading data to table default.test_part2 partition (x=null) from file:/user/hive/warehouse/test_part2/.hive-staging_hive_2015-06-02_11-27-21_770_3880518525165134918-1/-ext-10000
INFO  : 	 Time taken for load dynamic partitions : 532
INFO  : 	Loading partition {x=11-1031}
INFO  : 	Loading partition {x=11-1011}
INFO  : 	Loading partition {x=00-0000}
INFO  : 	Loading partition {x=11-1021}
INFO  : 	Loading partition {x=11-0000}
INFO  : 	 Time taken for adding to write entity : 1
INFO  : Partition default.test_part2{x=00-0000} stats: [numFiles=1, numRows=1, totalSize=16,
rawDataSize=15]
INFO  : Partition default.test_part2{x=11-0000} stats: [numFiles=1, numRows=1, totalSize=23,
rawDataSize=22]
INFO  : Partition default.test_part2{x=11-1011} stats: [numFiles=1, numRows=1, totalSize=17,
rawDataSize=16]
INFO  : Partition default.test_part2{x=11-1021} stats: [numFiles=1, numRows=1, totalSize=32,
rawDataSize=31]
INFO  : Partition default.test_part2{x=11-1031} stats: [numFiles=1, numRows=1, totalSize=12,
rawDataSize=11]
No rows affected (564.874 seconds)
0: jdbc:hive2://localhost:10000> insert overwrite table test_part2 partition (x) select
description , code from jspsrcsmall;
insert overwrite table test_part2 partition (x) 
 select description , code from jspsrcsmall;
INFO  : Number of reduce tasks is set to 0 since there's no reduce operator
INFO  : Job running in-process (local Hadoop)
INFO  : 2015-06-02 11:42:00,687 Stage-1 map = 0%,  reduce = 0%
INFO  : Ended Job = job_local1912867068_0011
INFO  : Stage-3 is selected by condition resolver.
INFO  : Stage-2 is filtered out by condition resolver.
INFO  : Stage-4 is filtered out by condition resolver.
INFO  : Moving data to: file:/user/hive/warehouse/test_part2/.hive-staging_hive_2015-06-02_11-41-19_090_6995116246086250594-1/-ext-10000
from file:/user/hive/warehouse/test_part2/.hive-staging_hive_2015-06-02_11-41-19_090_6995116246086250594-1/-ext-10002
INFO  : Loading data to table default.test_part2 partition (x=null) from file:/user/hive/warehouse/test_part2/.hive-staging_hive_2015-06-02_11-41-19_090_6995116246086250594-1/-ext-10000
INFO  : 	 Time taken for load dynamic partitions : 535
INFO  : 	Loading partition {x=11-1011}
INFO  : 	Loading partition {x=11-1031}
INFO  : 	Loading partition {x=11-1021}
INFO  : 	Loading partition {x=00-0000}
INFO  : 	Loading partition {x=11-0000}
INFO  : 	 Time taken for adding to write entity : 1
No rows affected (42.263 seconds)
{noformat}


> alter partition should check for "hive.stats.autogather" in hiveConf
> --------------------------------------------------------------------
>
>                 Key: HIVE-8955
>                 URL: https://issues.apache.org/jira/browse/HIVE-8955
>             Project: Hive
>          Issue Type: Improvement
>          Components: Metastore
>    Affects Versions: 0.13.1
>            Reporter: Pankit Thapar
>            Assignee: Yongzhi Chen
>
> When alter partition code path is triggered, it should check for the flag "hive.stats.autogather",
if it is true, then only updateStats else skip them.
> This is done in append_partition code flow. 
> Is there any specific reason the alter_partition does not respect this conf variable?
> //code snippet : HiveMetastore.java 
>  private Partition append_partition_common(RawStore ms, String dbName, String tableName,
>         List<String> part_vals, EnvironmentContext envContext) throws InvalidObjectException,
>         AlreadyExistsException, MetaException {
> ...
> ....
>         if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVESTATSAUTOGATHER) &&
>             !MetaStoreUtils.isView(tbl)) {
>           MetaStoreUtils.updatePartitionStatsFast(part, wh, madeDir);
>         }
> ...
> ...
> }
> The above code snippet checks for the variable but this same check is absent in 
> //code snippet : HiveAlterHandler.java 
> public Partition alterPartition(final RawStore msdb, Warehouse wh, final String dbname,
>       final String name, final List<String> part_vals, final Partition new_part)
>       throws InvalidOperationException, InvalidObjectException, AlreadyExistsException,
>       MetaException {
> ....
> ...
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message