Return-Path: <dev-return-12024-archive-asf-public=cust-asf.ponee.io@reef.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 32192200B41
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Thu, 23 Jun 2016 05:56:18 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id 30A7C160A64; Thu, 23 Jun 2016 03:56:18 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 52290160A36
	for <archive-asf-public@cust-asf.ponee.io>; Thu, 23 Jun 2016 05:56:17 +0200 (CEST)
Received: (qmail 25420 invoked by uid 500); 23 Jun 2016 03:56:16 -0000
Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@reef.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@reef.apache.org>
List-Post: <mailto:dev@reef.apache.org>
List-Id: <dev.reef.apache.org>
Reply-To: dev@reef.apache.org
Delivered-To: mailing list dev@reef.apache.org
Received: (qmail 25405 invoked by uid 99); 23 Jun 2016 03:56:16 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2016 03:56:16 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 3E1B72C14E1
	for <dev@reef.apache.org>; Thu, 23 Jun 2016 03:56:16 +0000 (UTC)
Date: Thu, 23 Jun 2016 03:56:16 +0000 (UTC)
From: "Dhruv Mahajan (JIRA)" <jira@apache.org>
To: dev@reef.apache.org
Message-ID: <JIRA.12980094.1466144495000.5437.1466654176141@Atlassian.JIRA>
In-Reply-To: <JIRA.12980094.1466144495000@Atlassian.JIRA>
References: <JIRA.12980094.1466144495000@Atlassian.JIRA> <JIRA.12980094.1466144495399@arcas>
Subject: [jira] [Commented] (REEF-1456) Develop distributed counters
 framework in REEF
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Thu, 23 Jun 2016 03:56:18 -0000


    [ https://issues.apache.org/jira/browse/REEF-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345682#comment-15345682 ] 

Dhruv Mahajan commented on REEF-1456:
-------------------------------------

{code}
    /// <summary>
    /// Interface for the distributed counter. The implementation will not be thread-safe.
    /// </summary>
    /// <typeparam name="T">Type of counter: typical values: int, float, double</typeparam>
    public interface IDistributedCounter<T>
    {
        /// <summary>
        /// Unique name of the counter. No two cunters should have same names.
        /// </summary>
        string Name {get; private set;}

        /// <summary>
        /// Value of the counter
        /// </summary>
        T Counter {get; private set;}

        /// <summary>
        /// Updates the counter by value. For += update these are simple 
        /// increments while for Most recently updated (MRU) case this is the latest 
        /// value. Actually for MRU case T itself is a structure with the value and 
        /// timestamp.
        /// </summary>
        /// <typeparam name="T">Type of counter.</typeparam>
        /// <param name="value">Value to update the counter.</param>
        void Update(ref T value);

        /// <summary>
        /// Whether the value of the counter is default or not. Useful to 
        /// check during serialization to determine whether to send it or not.
        /// </summary>
        /// <returns>True if value os default, False otherwise.</returns>
        bool EqualsDefault();

        /// <summary>
        /// Resets the counter to default value.
        /// </summary>
        void Reset();

        /// <summary>
        /// Serialize the counter. It will contain all the information to create 
        /// a new counter by reflection on driver or update the existing counter.
        /// </summary>
        /// <param name="writer">writer to write the serialized data</param>
        void Serialize(IDataWriter writer);

        /// <summary>
        /// Deseralizes the update and then update the counter. This function will be 
        /// used at the driver side to get the counter value sent from tasks and then 
        /// update the existing value.
        /// </summary>
        /// <param name="reader"></param>
        void DeSerializeValueAndUpdate(IDataReader reader);
    }

    /// <summary>
    /// Client side that manages counters. This interface will be injected on evaluators 
    /// either as part of task or context. A default implementation would be provided.
    /// This class will be thread-safe in a sense that updates to a counter will be 
    /// applied sequentially and at a given time we can enter only one of the functions.
    /// </summary>
    public interface IDistributedCountersClient
    {
        /// <summary>
        /// Registers the new counter.
        /// </summary>
        /// <typeparam name="T">Type of counter.</typeparam>
        /// <param name="counter">Counter class.</param>
        void Register<T>(IDistributedCounter<T> counter);
        
        /// <summary>
        /// Update counter by name.
        /// </summary>
        /// <typeparam name="T">Type of counter</typeparam>
        /// <param name="counterName">Name of counter</param>
        /// <param name="value">Value with which the counter should be updated</param>
        void Update<T>(string counterName, ref T value);

        /// <summary>
        /// Resets counter by name to default vlaue.
        /// </summary>
        /// <typeparam name="T">Type of counter</typeparam>
        /// <param name="counterName">Name of the counter</param>
        void Reset<T>(string counterName);

        /// <summary>
        /// Resets all the counter.
        /// </summary>
        void ResetAll();

        /// <summary>
        /// Serialize all the counters with non-default values
        /// </summary>
        /// <param name="writer">Writer to which serialized counters are written.</param>
        void SerializeCounters(IDataWriter writer);
    }

    /// <summary>
    /// Server or driver side that manages counters. This interface will be used on driver side. 
    /// A default implementation would be provided. This class will be threadsafe in a sense that 
    /// that at any given point of time we can enter only one of the functions.
    /// </summary>
    public interface IDistributedCountersServer
    {        
        /// <summary>
        /// Serializes the counters by providing the names. This function will be called by driver 
        /// when REEF client asks for the values of some counters.
        /// </summary>
        /// <param name="counterNames">The name of counters that need to be serialized</param>
        /// <param name="writer">Writer to write the serialized counters.</param>
        void SerializeCountersByName(IEnumerable<string> counterNames, IDataWriter writer);

        /// <summary>
        /// Enumerates the name of the counters currently available. This function will be called 
        /// by driver when REEF client asks for the names of existing counters.
        /// </summary>
        /// <returns></returns>
        IEnumerable<string> EnumerateCounterNames();

        /// <summary>
        /// Deserializes the counter from stream and use them to update existing counters.
        /// Creates new counter by reflection if it does not exist. 
        /// </summary>
        /// <param name="reader">Reader from which to read the data.</param>
        void DeSerializeAndUpdateCounters(IDataReader reader);
    }
{code}

> Develop distributed counters framework in REEF
> ----------------------------------------------
>
>                 Key: REEF-1456
>                 URL: https://issues.apache.org/jira/browse/REEF-1456
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>         Environment: C#
>            Reporter: Dhruv Mahajan
>
> The aim of this JIRA is to develop distributed counters framework in REEF. Each task can emit pairs of {counter name, incremental value} which are aggregated and sent to driver, which can then aggregate them from all the tasks/ evaluators. The aggregation strategy can be simple addition, most recently used, etc.  Via these counters, we can implement some common metrics for ML - like amount (bytes) of data read, current loss function value etc.
> We will develop interfaces for Counters for evaluators and driver along with default implementations and aggregation strategies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)