phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Soldatov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3871) Incremental stats collection
Date Mon, 22 May 2017 23:05:04 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020365#comment-16020365
] 

Sergey Soldatov commented on PHOENIX-3871:
------------------------------------------

We collect statistic on all compactions that have COMPACT_DROP_DELETES. Those not only major
compactions, but also minor compactions if one of default compaction policies are used ( RatioBasedCompactionPolicy
sets the compaction as major if all storefile candidates get into the compaction). Running
statistic collection on upserts sounds like an overkill. 

> Incremental stats collection
> ----------------------------
>
>                 Key: PHOENIX-3871
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3871
>             Project: Phoenix
>          Issue Type: Improvement
>            Reporter: Eli Levine
>
> Phoenix automatically gathers statistics at [major compaction time|http://phoenix.apache.org/update_statistics.html].
While this is useful and accurate, it also means that statistics can become stale due to the
infrequency of major compactions (can be days between major compactions), reducing their usefulness.

> This jira asks the question: Is it possible for Phoenix to collects statistics at a more
granular level, say for every (or a sampling of) UPSERT, or minor compaction. Since statistics
are always approximations, it is OK for this incremental approach to not be 100% accurate.
> The current stats collection mechanism at major compaction time should be kept to accurately
"fix up" stats at major compaction time.
> [~jamestaylor], FYI. We talked about this in person a few weeks ago. Creating this Jira
for posterity. Please add anything that I missed. Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message