ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Persistent Distributed Store Metrics
Date Tue, 30 May 2017 18:58:39 GMT
> Also, I see your argument about separating memory from disk metrics, but I
> think it makes sense to combine them, because it is hard to separate them
> from user standpoint. Let's call them IgniteStorageMetrics or something of
> the kind.

The issue is that we can’t combine all of them because these

>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>>    public double getPageEvictionRate();
>> 
>>    public double getPageMissRate();
>> 
>>    public long getDirtyPages();
>> 
>>    public long getPagesOnDisk();
>> }


are calculated per memory policy (memory region) while these

>> public interface PersistenceStoreMetrics {
>>    //WAL-related metrics
>>    public double getWalLoggingRate();
>> 
>>    public int getWalArchiveSegments();
>> 
>>    public double getWalFsyncAverage();
>> 
>>    //checkpoint-related metrics
>>    public double getCpTime();
>> 
>>    public double getCpFsyncTime();
>> 
>>    public long getCpTotalPages();
>> 
>>    public long[] getCpPagesByType();
>> 
>>    public long getCpCopiedOnWritePages();
>> }

are global.

 
—
Denis

> On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <dsetrakyan@apache.org> wrote:
> 
> Several questions:
> 
>   1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should be
>   consistent with other time metrics, and should be called
>   "getWalFsyncAverageTime()"
>   2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
>   3. Also, how about WAL total pages count or total size, whichever one is
>   easier?
>   4. How about the WAL size that has not been checkpointed yet?
>   5. How about WAL history size (the number of WAL files that we are
>   rotating through)?
> 
> Also, I see your argument about separating memory from disk metrics, but I
> think it makes sense to combine them, because it is hard to separate them
> from user standpoint. Let's call them IgniteStorageMetrics or something of
> the kind.
> 
> D.
> 
> 
> On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <sergey.chugunov@gmail.com>
> wrote:
> 
>> Hello Denis,
>> 
>> I have an idea of how we could clarify proposed above approach to
>> collecting metrics for PDS.
>> 
>> I suggest to split all metrics into two major groups (disk-related and
>> memory-related) and group them in the following interfaces:
>> 
>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>>    public double getPageEvictionRate();
>> 
>>    public double getPageMissRate();
>> 
>>    public long getDirtyPages();
>> 
>>    public long getPagesOnDisk();
>> }
>> 
>> public interface PersistenceStoreMetrics {
>>    //WAL-related metrics
>>    public double getWalLoggingRate();
>> 
>>    public int getWalArchiveSegments();
>> 
>>    public double getWalFsyncAverage();
>> 
>>    //checkpoint-related metrics
>>    public double getCpTime();
>> 
>>    public double getCpFsyncTime();
>> 
>>    public long getCpTotalPages();
>> 
>>    public long[] getCpPagesByType();
>> 
>>    public long getCpCopiedOnWritePages();
>> }
>> 
>> From my standpoint it is important to distinguish between PDS metrics that
>> are related mostly to memory management (from the first interface) and
>> metrics bound to disk operations (from the second interface).
>> 
>> First group of metrics is useful when collected per Memory Policy. It
>> doesn't provide a lot of information when collected across many Memory
>> Policies.
>> 
>> Second group is now global (this may change in the future if Persistence
>> Policies are introduced) and allows to easily spot any high-level issues
>> with durable storage configuration or performance. More specific questions
>> like "how this Memory Policy loads a durable storage subsystem?" may be
>> deduced from PersistenceMemoryMetrics indicators.
>> 
>> Denis, does it make sense for you?
>> 
>> Any other ideas/questions/improvements or metrics to add to these
>> interfaces?
>> 
>> I would like to discuss all the details here before creating any tickets
>> for the change.
>> 
>> Thanks,
>> Sergey.
>> 
>> 
>> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dmagda@apache.org> wrote:
>> 
>>> Dmitriy,
>>> 
>>> I have some high level concerns.
>>> 
>>> Presently, Ignite already exposes memory/data related metrics via
>>> MemoryMetrics and CacheMetrics interface.
>>> 
>>> Looking at PersistentStoreMetrics interface I can’t get why methods like
>>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT
>>> in MemoryMetrics interface.
>>> 
>>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
>>> contains a big subset of the methods from PersistentStoreMetric. There
>> just
>>> should be some API call that either returns global metrics or cache
>>> specific ones. Plus, some of the methods might be added to existing
>>> CacheInterface. Why not.
>>> 
>>> I prefer to address this first. Guys, any other thoughts on this?
>>> 
>>> —
>>> Denis
>>> 
>>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
>>> dmitriy.govorukhin@gmail.com> wrote:
>>>> 
>>>> Folk,
>>>> 
>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>> provide ability to store data on disk. Let's discuss what type of
>>>> metrics we need for this?
>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>> and global metrics which will be aggregate all metrics.
>>>> 
>>>> I did sketch.
>>>> 
>>>> PersistentStoreMetrics.java
>>>> 
>>>> public interface PersistentStoreMetrics {
>>>> 
>>>>   // Global metrics.
>>>> 
>>>>   public long getMemorySize();
>>>> 
>>>>   public long getDiskSize();
>>>> 
>>>>   public long getPagesInMemory();
>>>> 
>>>>   public long getPagesSizeInMemory();
>>>> 
>>>>   public long getPagesOnDisk();
>>>> 
>>>>   public long getPagesSizeOnDisk();
>>>> 
>>>>   public long getFreePages();
>>>> 
>>>>   public long getFreePagesSize();
>>>> 
>>>>   public long getDirtyPages();
>>>> 
>>>>   public long getDirtyPagesSize();
>>>> 
>>>>   public long walLog();
>>>> 
>>>>   public long walLogSize();
>>>> 
>>>>   // Frequency.
>>>> 
>>>>   public long getPagesRead();
>>>> 
>>>>   public long getPagesWrite();
>>>> 
>>>>   public long getFsync();
>>>> 
>>>>   public long getWal();
>>>> 
>>>>   public long getAverageWalFsyncTime();
>>>> 
>>>>   // Per cache.
>>>> 
>>>>   public PersistentStoreCacheMetrics cache(String name);
>>>> 
>>>>   public PersistentStoreCacheMetrics cache(int cacheId);
>>>> 
>>>>   // For last checkpoint.
>>>> 
>>>>   public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>> }
>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>> 
>>>> PersistentStoreCacheMetrics.java
>>>> 
>>>> public interface PersistentStoreCacheMetrics {
>>>> 
>>>>   public String name();
>>>> 
>>>>   public double getFillFactor();
>>>> 
>>>>   public double getFillFactor(int part);
>>>> 
>>>>   public long getMemorySize();
>>>> 
>>>>   public long getDiskSize();
>>>> 
>>>>   public long getPagesInMemory();
>>>> 
>>>>   public long getPagesSizeInMemory();
>>>> 
>>>>   public long getPagesOnDisk();
>>>> 
>>>>   public long getPagesSizeOnDisk();
>>>> 
>>>>   public long getFreePages();
>>>> 
>>>>   public long getFreePagesSize();
>>>> 
>>>>   public long getDirtyPages();
>>>> 
>>>>   public long getDirtyPagesSize();
>>>> 
>>>>   public long getPagesRead();
>>>> 
>>>>   public long getPagesWritten();
>>>> }
>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>> 
>>>> PersistentStoreCheckpointMetrics.java
>>>> 
>>>> public interface PersistentStoreCheckpointMetrics {
>>>> 
>>>>   public long getTotalPages();
>>>> 
>>>>   //TODO Page type is internal?
>>>>   public long[] pagesType();
>>>> 
>>>>   public long getExecutingTime();
>>>> 
>>>>   public long getFsyncTime();
>>>> 
>>>>   public long getPagesCopyOnWrite();
>>>> }
>>> 
>>> 
>> 


Mime
View raw message