mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Bannier <benjamin.bann...@mesosphere.io>
Subject Re: [Design Doc] Native Support for Prometheus Metrics
Date Sat, 09 Sep 2017 12:29:24 GMT
Hi James,

I'd like to make a longer comment here to make it easier to discuss.

> [...]
> 
> Note the proposal to alter how Timer metrics are exposed in an incompatible
> way (I argue this is OK because you can't really make use of these metrics
> now).

I am not sure I follow your argument around `Timer`. It is similar to a gauge
caching the last value and an associated statistics calculated from a time series.

I have never used Prometheus, but a brief look at the Prometheus
docs seems to suggest that a `Timer` could be mapped onto a Prometheus summary
type with minimal modifications (namely, by adding a `sum` value that you
propose as sole replacement).

I believe that exposing statistics is useful, and moving all `Timer` metrics to
counters (cumulative value and number of samples) would leads to information
loss.

Since most of your criticism of `Timer` is about it its associated statistics,
maybe we can make fixes to libprocess' `TimeSeries` and the derived
`Statistics` to make them more usable. Right now `Statistics` seems to be more
apt for dealing with timing measurements where one probably worries more about
the long tail of the distribution (it only exposes the median and higher
percentiles). It seems that if one would e.g., make the exposed percentiles
configurable, it should be possible to expose a useful characterization of the
underlying distribution (think: box plot). It might be that one would need to
revisit how `TimeSeries` sparsifies older data to make sure the quantiles we
expose are meaningful.

> First, note that the “allocator/mesos/allocation_run_ms/count” sample is not
> useful at all. It has the semantics of a saturating counter that saturates at
> the size of the bounded time series. To address this, there is another metric
> “allocator/mesos/allocation_runs”, which tracks the actual count of
> allocation runs (3161331.00 in this case). If you plot this counter over time
> (ie. as a rate), it will be zero for all time once it reaches saturation. In
> the case of allocation runs, this is almost all the time, since 1000
> allocations will be performed within a few hours.

While `count` is not a useful measure of the behavior of the measured datum, it
is critical to assess whether the derived statistic is meaningful (sample
size). Like you write, it becomes less interesting once enough data was
collected.

> Finally, while the derived statistics metrics can be informative, they are
> actually less expressive than a raw histogram would be. A raw histogram of
> timed values would allow an observer to distinguish cases where there are
> clear performance bands (e.g. when allocation times cluster at either 15ms or
> 200ms), but the percentile statistics obscure this information.

I would argue that is more a problem of `Statistics` only reporting percentiles
from the far out, large value tail. Would e.g., the reported percentiles be
placed more evenly it should be possible to recognize bands. After all
percentiles are just samples from the cumulative distribution from which one
can derived the underlying distribution (with some resolution) by taking a
derivative.

Note that a poorly binned and ranged histogram can obscure features as well.
Reporting percentiles/quantiles has the advantage of adapting to the data
automatically.


Cheers,

Benjamin

Mime
View raw message