ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Persistent Distributed Store Metrics
Date Tue, 30 May 2017 19:29:01 GMT
It looks to me like all global metrics have to do with WAL. Why not call
them IgniteWalMetrics then?

On Tue, May 30, 2017 at 11:58 AM, Denis Magda <dmagda@apache.org> wrote:

> > Also, I see your argument about separating memory from disk metrics, but
> I
> > think it makes sense to combine them, because it is hard to separate them
> > from user standpoint. Let's call them IgniteStorageMetrics or something
> of
> > the kind.
>
> The issue is that we can’t combine all of them because these
>
> >> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> >>    public double getPageEvictionRate();
> >>
> >>    public double getPageMissRate();
> >>
> >>    public long getDirtyPages();
> >>
> >>    public long getPagesOnDisk();
> >> }
>
>
> are calculated per memory policy (memory region) while these
>
> >> public interface PersistenceStoreMetrics {
> >>    //WAL-related metrics
> >>    public double getWalLoggingRate();
> >>
> >>    public int getWalArchiveSegments();
> >>
> >>    public double getWalFsyncAverage();
> >>
> >>    //checkpoint-related metrics
> >>    public double getCpTime();
> >>
> >>    public double getCpFsyncTime();
> >>
> >>    public long getCpTotalPages();
> >>
> >>    public long[] getCpPagesByType();
> >>
> >>    public long getCpCopiedOnWritePages();
> >> }
>
> are global.
>
>
> —
> Denis
>
> > On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
> >
> > Several questions:
> >
> >   1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should
> be
> >   consistent with other time metrics, and should be called
> >   "getWalFsyncAverageTime()"
> >   2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
> >   3. Also, how about WAL total pages count or total size, whichever one
> is
> >   easier?
> >   4. How about the WAL size that has not been checkpointed yet?
> >   5. How about WAL history size (the number of WAL files that we are
> >   rotating through)?
> >
> > Also, I see your argument about separating memory from disk metrics, but
> I
> > think it makes sense to combine them, because it is hard to separate them
> > from user standpoint. Let's call them IgniteStorageMetrics or something
> of
> > the kind.
> >
> > D.
> >
> >
> > On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> > wrote:
> >
> >> Hello Denis,
> >>
> >> I have an idea of how we could clarify proposed above approach to
> >> collecting metrics for PDS.
> >>
> >> I suggest to split all metrics into two major groups (disk-related and
> >> memory-related) and group them in the following interfaces:
> >>
> >> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> >>    public double getPageEvictionRate();
> >>
> >>    public double getPageMissRate();
> >>
> >>    public long getDirtyPages();
> >>
> >>    public long getPagesOnDisk();
> >> }
> >>
> >> public interface PersistenceStoreMetrics {
> >>    //WAL-related metrics
> >>    public double getWalLoggingRate();
> >>
> >>    public int getWalArchiveSegments();
> >>
> >>    public double getWalFsyncAverage();
> >>
> >>    //checkpoint-related metrics
> >>    public double getCpTime();
> >>
> >>    public double getCpFsyncTime();
> >>
> >>    public long getCpTotalPages();
> >>
> >>    public long[] getCpPagesByType();
> >>
> >>    public long getCpCopiedOnWritePages();
> >> }
> >>
> >> From my standpoint it is important to distinguish between PDS metrics
> that
> >> are related mostly to memory management (from the first interface) and
> >> metrics bound to disk operations (from the second interface).
> >>
> >> First group of metrics is useful when collected per Memory Policy. It
> >> doesn't provide a lot of information when collected across many Memory
> >> Policies.
> >>
> >> Second group is now global (this may change in the future if Persistence
> >> Policies are introduced) and allows to easily spot any high-level issues
> >> with durable storage configuration or performance. More specific
> questions
> >> like "how this Memory Policy loads a durable storage subsystem?" may be
> >> deduced from PersistenceMemoryMetrics indicators.
> >>
> >> Denis, does it make sense for you?
> >>
> >> Any other ideas/questions/improvements or metrics to add to these
> >> interfaces?
> >>
> >> I would like to discuss all the details here before creating any tickets
> >> for the change.
> >>
> >> Thanks,
> >> Sergey.
> >>
> >>
> >> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dmagda@apache.org> wrote:
> >>
> >>> Dmitriy,
> >>>
> >>> I have some high level concerns.
> >>>
> >>> Presently, Ignite already exposes memory/data related metrics via
> >>> MemoryMetrics and CacheMetrics interface.
> >>>
> >>> Looking at PersistentStoreMetrics interface I can’t get why methods
> like
> >>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are
> NOT
> >>> in MemoryMetrics interface.
> >>>
> >>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
> >>> contains a big subset of the methods from PersistentStoreMetric. There
> >> just
> >>> should be some API call that either returns global metrics or cache
> >>> specific ones. Plus, some of the methods might be added to existing
> >>> CacheInterface. Why not.
> >>>
> >>> I prefer to address this first. Guys, any other thoughts on this?
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
> >>> dmitriy.govorukhin@gmail.com> wrote:
> >>>>
> >>>> Folk,
> >>>>
> >>>> As you know, ignite 2.1 will contain new module (pds), it will be
> >>>> provide ability to store data on disk. Let's discuss what type of
> >>>> metrics we need for this?
> >>>> I think it must be metrics per memory policy, per cache, checkpoint,
> >>>> and global metrics which will be aggregate all metrics.
> >>>>
> >>>> I did sketch.
> >>>>
> >>>> PersistentStoreMetrics.java
> >>>>
> >>>> public interface PersistentStoreMetrics {
> >>>>
> >>>>   // Global metrics.
> >>>>
> >>>>   public long getMemorySize();
> >>>>
> >>>>   public long getDiskSize();
> >>>>
> >>>>   public long getPagesInMemory();
> >>>>
> >>>>   public long getPagesSizeInMemory();
> >>>>
> >>>>   public long getPagesOnDisk();
> >>>>
> >>>>   public long getPagesSizeOnDisk();
> >>>>
> >>>>   public long getFreePages();
> >>>>
> >>>>   public long getFreePagesSize();
> >>>>
> >>>>   public long getDirtyPages();
> >>>>
> >>>>   public long getDirtyPagesSize();
> >>>>
> >>>>   public long walLog();
> >>>>
> >>>>   public long walLogSize();
> >>>>
> >>>>   // Frequency.
> >>>>
> >>>>   public long getPagesRead();
> >>>>
> >>>>   public long getPagesWrite();
> >>>>
> >>>>   public long getFsync();
> >>>>
> >>>>   public long getWal();
> >>>>
> >>>>   public long getAverageWalFsyncTime();
> >>>>
> >>>>   // Per cache.
> >>>>
> >>>>   public PersistentStoreCacheMetrics cache(String name);
> >>>>
> >>>>   public PersistentStoreCacheMetrics cache(int cacheId);
> >>>>
> >>>>   // For last checkpoint.
> >>>>
> >>>>   public PersistentStoreCheckpointMetrics getLastCheckPoint();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCacheMetrics.java
> >>>>
> >>>> public interface PersistentStoreCacheMetrics {
> >>>>
> >>>>   public String name();
> >>>>
> >>>>   public double getFillFactor();
> >>>>
> >>>>   public double getFillFactor(int part);
> >>>>
> >>>>   public long getMemorySize();
> >>>>
> >>>>   public long getDiskSize();
> >>>>
> >>>>   public long getPagesInMemory();
> >>>>
> >>>>   public long getPagesSizeInMemory();
> >>>>
> >>>>   public long getPagesOnDisk();
> >>>>
> >>>>   public long getPagesSizeOnDisk();
> >>>>
> >>>>   public long getFreePages();
> >>>>
> >>>>   public long getFreePagesSize();
> >>>>
> >>>>   public long getDirtyPages();
> >>>>
> >>>>   public long getDirtyPagesSize();
> >>>>
> >>>>   public long getPagesRead();
> >>>>
> >>>>   public long getPagesWritten();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCheckpointMetrics.java
> >>>>
> >>>> public interface PersistentStoreCheckpointMetrics {
> >>>>
> >>>>   public long getTotalPages();
> >>>>
> >>>>   //TODO Page type is internal?
> >>>>   public long[] pagesType();
> >>>>
> >>>>   public long getExecutingTime();
> >>>>
> >>>>   public long getFsyncTime();
> >>>>
> >>>>   public long getPagesCopyOnWrite();
> >>>> }
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message