reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruv Mahajan (JIRA)" <>
Subject [jira] [Commented] (REEF-1456) Develop distributed counters framework in REEF
Date Thu, 23 Jun 2016 23:41:16 GMT


Dhruv Mahajan commented on REEF-1456:

[~markus.weimer] Another question is that how does driver know the Codecs for the the Value
type {{T}} in the counter? For this I am thinking of making {{IDistributedCounter<T>}}
derive from base classes {{IStreamingCodec<T>}} and {{IBaseCounter}}. Then we will have
only one implementation of counter code as: {{CounterCodec : IStreamingCodec<IBaseCounter>}}
which will also encode the actual counter type. Then on driver side {{CounterCodec}} will
create the desired counter using this Type and reflection.

> Develop distributed counters framework in REEF
> ----------------------------------------------
>                 Key: REEF-1456
>                 URL:
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>         Environment: C#
>            Reporter: Dhruv Mahajan
> The aim of this JIRA is to develop distributed counters framework in REEF. Each task
can emit pairs of {counter name, incremental value} which are aggregated and sent to driver,
which can then aggregate them from all the tasks/ evaluators. The aggregation strategy can
be simple addition, most recently used, etc.  Via these counters, we can implement some common
metrics for ML - like amount (bytes) of data read, current loss function value etc.
> We will develop interfaces for Counters for evaluators and driver along with default
implementations and aggregation strategies.

This message was sent by Atlassian JIRA

View raw message