reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruv Mahajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1456) Develop distributed counters framework in REEF
Date Mon, 04 Jul 2016 06:30:11 GMT

    [ https://issues.apache.org/jira/browse/REEF-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360895#comment-15360895
] 

Dhruv Mahajan commented on REEF-1456:
-------------------------------------

[~markus.weimer] [~anupam128] after offline discussion with [~anupam128], we sort of converged
that we will support {{ulong}} and {{double}} as the only types... They cover all the common
types (float, int, bool, byte, double, long). [~anupam128] was of the opinion that {{ulong}}
alone is sufficient and application/user can multiply double quantities (loss for example)
by {{10^n}} and then store in {{ulong}}. But I am of the opinion that this loss in precision
might be crucial for reporting some metrics related to ML especially if the user himself is
a ML person. So I have decided to go with both for the time being. Anyways, later if we want
to support only one, it should be a trivial change. 

> Develop distributed counters framework in REEF
> ----------------------------------------------
>
>                 Key: REEF-1456
>                 URL: https://issues.apache.org/jira/browse/REEF-1456
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>         Environment: C#
>            Reporter: Dhruv Mahajan
>
> The aim of this JIRA is to develop distributed counters framework in REEF. Each task
can emit pairs of {counter name, incremental value} which are aggregated and sent to driver,
which can then aggregate them from all the tasks/ evaluators. The aggregation strategy can
be simple addition, most recently used, etc.  Via these counters, we can implement some common
metrics for ML - like amount (bytes) of data read, current loss function value etc.
> We will develop interfaces for Counters for evaluators and driver along with default
implementations and aggregation strategies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message