hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12411) Remove counter based stats collection mechanism
Date Wed, 25 Nov 2015 01:46:11 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025967#comment-15025967
] 

Lefty Leverenz commented on HIVE-12411:
---------------------------------------

Doc note:  This changes *hive.stats.dbclass* (removing counter as a value) and removes *hive.stats.key.prefix.reserve.length*
so the wiki needs to be updated for release 2.0.0.

* [Configuration Properties -- hive.stats.dbclass | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.dbclass]
* [Configuration Properties -- hive.stats.key.prefix.reserve.length | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.key.prefix.reserve.length]

The Statistics doc does not mention counter-based stats so no update is required, although
an explanation of collection mechanisms would be a helpful addition.   *hive.stats.dbclass*
is discussed in the Usage section.

* [Statistics in Hive | https://cwiki.apache.org/confluence/display/Hive/StatsDev]
** [Implementation | https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-Implementation]
** [Usage | https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-Usage]

> Remove counter based stats collection mechanism
> -----------------------------------------------
>
>                 Key: HIVE-12411
>                 URL: https://issues.apache.org/jira/browse/HIVE-12411
>             Project: Hive
>          Issue Type: Task
>          Components: Statistics
>    Affects Versions: 1.2.0, 1.2.1
>            Reporter: Pengcheng Xiong
>            Assignee: Pengcheng Xiong
>              Labels: TODOC2.0
>             Fix For: 2.0.0
>
>         Attachments: HIVE-12411.01.patch, HIVE-12411.02.patch
>
>
> Following HIVE-12005, HIVE-12164, we have removed jdbc and hbase stats collection mechanism.
Now we are targeting counter based stats collection mechanism. The main advantages are as
follows (1) counter based stats has limitation on the length of the counter itself, if it
is too long, MD5 will be applied. (2) when there are a large number of partitions and columns,
we need to create a large number of counters in memory. This will put a heavy load on the
M/R AM or Tez AM etc. FS based stats will do a better job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message