hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharat Viswanadham (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-816) Create OM metrics for bucket, volume, keys
Date Thu, 15 Nov 2018 18:30:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688482#comment-16688482

Bharat Viswanadham commented on HDDS-816:

Hi [~elek]

The reason why we have not gone with rocksdb estimate number of keys is, like if we take this
value during OM start, and DB compaction is not done, still we see the wrong value. Suppose,
if we have a background thread, which takes this values for every configured interval, it
causes an impression to admin that key count is varying continously somehow, without any
user action (like deletion, creation), and it will cause a panic to admin. And also the estimates
key count can also be an issue with this approach because the calculation wibe estimatedate
when below are happening on rocksdb. (comments from code of rocksdb)

(1) there exist merge keys

(2) keys are directly overwritten

(3) deletion on non-existing keys


So, on our OM rocksb 2 and 3 can happen, so the estimate will be mostly wrong. Even if we
think after compaction we get this value, but after some time background thread gets the estimate
count value again and we will be wrong. And if we don't have background thread to update
at the frequent interval the behavior is unpredictable and it depends on rocksdb compaction

So, I think missing the interval if OM is having unclean shutdown will be okay. And atleast
we shall be accurate in the happy scenario.



> Create OM metrics for bucket, volume, keys
> ------------------------------------------
>                 Key: HDDS-816
>                 URL: https://issues.apache.org/jira/browse/HDDS-816
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Manager
>            Reporter: Bharat Viswanadham
>            Assignee: Bharat Viswanadham
>            Priority: Major
>         Attachments: HDDS-816.00.patch, HDDS-816.01.patch, HDDS-816.03.patch, Metrics
for number of volumes, buckets, keys.pdf, Proposed Approach.pdf
> This Jira is used to create the following metrics in Ozone manager.
>  # number of volumes 
>  # number of buckets
>  # number of keys

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message