reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruv Mahajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1456) Develop distributed counters framework in REEF
Date Thu, 23 Jun 2016 03:56:16 GMT

    [ https://issues.apache.org/jira/browse/REEF-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345682#comment-15345682
] 

Dhruv Mahajan commented on REEF-1456:
-------------------------------------

{code}
    /// <summary>
    /// Interface for the distributed counter. The implementation will not be thread-safe.
    /// </summary>
    /// <typeparam name="T">Type of counter: typical values: int, float, double</typeparam>
    public interface IDistributedCounter<T>
    {
        /// <summary>
        /// Unique name of the counter. No two cunters should have same names.
        /// </summary>
        string Name {get; private set;}

        /// <summary>
        /// Value of the counter
        /// </summary>
        T Counter {get; private set;}

        /// <summary>
        /// Updates the counter by value. For += update these are simple 
        /// increments while for Most recently updated (MRU) case this is the latest 
        /// value. Actually for MRU case T itself is a structure with the value and 
        /// timestamp.
        /// </summary>
        /// <typeparam name="T">Type of counter.</typeparam>
        /// <param name="value">Value to update the counter.</param>
        void Update(ref T value);

        /// <summary>
        /// Whether the value of the counter is default or not. Useful to 
        /// check during serialization to determine whether to send it or not.
        /// </summary>
        /// <returns>True if value os default, False otherwise.</returns>
        bool EqualsDefault();

        /// <summary>
        /// Resets the counter to default value.
        /// </summary>
        void Reset();

        /// <summary>
        /// Serialize the counter. It will contain all the information to create 
        /// a new counter by reflection on driver or update the existing counter.
        /// </summary>
        /// <param name="writer">writer to write the serialized data</param>
        void Serialize(IDataWriter writer);

        /// <summary>
        /// Deseralizes the update and then update the counter. This function will be 
        /// used at the driver side to get the counter value sent from tasks and then 
        /// update the existing value.
        /// </summary>
        /// <param name="reader"></param>
        void DeSerializeValueAndUpdate(IDataReader reader);
    }

    /// <summary>
    /// Client side that manages counters. This interface will be injected on evaluators 
    /// either as part of task or context. A default implementation would be provided.
    /// This class will be thread-safe in a sense that updates to a counter will be 
    /// applied sequentially and at a given time we can enter only one of the functions.
    /// </summary>
    public interface IDistributedCountersClient
    {
        /// <summary>
        /// Registers the new counter.
        /// </summary>
        /// <typeparam name="T">Type of counter.</typeparam>
        /// <param name="counter">Counter class.</param>
        void Register<T>(IDistributedCounter<T> counter);
        
        /// <summary>
        /// Update counter by name.
        /// </summary>
        /// <typeparam name="T">Type of counter</typeparam>
        /// <param name="counterName">Name of counter</param>
        /// <param name="value">Value with which the counter should be updated</param>
        void Update<T>(string counterName, ref T value);

        /// <summary>
        /// Resets counter by name to default vlaue.
        /// </summary>
        /// <typeparam name="T">Type of counter</typeparam>
        /// <param name="counterName">Name of the counter</param>
        void Reset<T>(string counterName);

        /// <summary>
        /// Resets all the counter.
        /// </summary>
        void ResetAll();

        /// <summary>
        /// Serialize all the counters with non-default values
        /// </summary>
        /// <param name="writer">Writer to which serialized counters are written.</param>
        void SerializeCounters(IDataWriter writer);
    }

    /// <summary>
    /// Server or driver side that manages counters. This interface will be used on driver
side. 
    /// A default implementation would be provided. This class will be threadsafe in a sense
that 
    /// that at any given point of time we can enter only one of the functions.
    /// </summary>
    public interface IDistributedCountersServer
    {        
        /// <summary>
        /// Serializes the counters by providing the names. This function will be called by
driver 
        /// when REEF client asks for the values of some counters.
        /// </summary>
        /// <param name="counterNames">The name of counters that need to be serialized</param>
        /// <param name="writer">Writer to write the serialized counters.</param>
        void SerializeCountersByName(IEnumerable<string> counterNames, IDataWriter writer);

        /// <summary>
        /// Enumerates the name of the counters currently available. This function will be
called 
        /// by driver when REEF client asks for the names of existing counters.
        /// </summary>
        /// <returns></returns>
        IEnumerable<string> EnumerateCounterNames();

        /// <summary>
        /// Deserializes the counter from stream and use them to update existing counters.
        /// Creates new counter by reflection if it does not exist. 
        /// </summary>
        /// <param name="reader">Reader from which to read the data.</param>
        void DeSerializeAndUpdateCounters(IDataReader reader);
    }
{code}

> Develop distributed counters framework in REEF
> ----------------------------------------------
>
>                 Key: REEF-1456
>                 URL: https://issues.apache.org/jira/browse/REEF-1456
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>         Environment: C#
>            Reporter: Dhruv Mahajan
>
> The aim of this JIRA is to develop distributed counters framework in REEF. Each task
can emit pairs of {counter name, incremental value} which are aggregated and sent to driver,
which can then aggregate them from all the tasks/ evaluators. The aggregation strategy can
be simple addition, most recently used, etc.  Via these counters, we can implement some common
metrics for ML - like amount (bytes) of data read, current loss function value etc.
> We will develop interfaces for Counters for evaluators and driver along with default
implementations and aggregation strategies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message