Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EA57D200B26 for ; Mon, 27 Jun 2016 23:46:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E7782160A5B; Mon, 27 Jun 2016 21:46:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 53DB7160A63 for ; Mon, 27 Jun 2016 23:46:58 +0200 (CEST) Received: (qmail 18713 invoked by uid 500); 27 Jun 2016 21:46:57 -0000 Mailing-List: contact dev-help@reef.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@reef.apache.org Delivered-To: mailing list dev@reef.apache.org Received: (qmail 18629 invoked by uid 99); 27 Jun 2016 21:46:57 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2016 21:46:57 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 553DC2C1F5C for ; Mon, 27 Jun 2016 21:46:57 +0000 (UTC) Date: Mon, 27 Jun 2016 21:46:57 +0000 (UTC) From: "Dhruv Mahajan (JIRA)" To: dev@reef.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (REEF-1456) Develop distributed counters framework in REEF MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 27 Jun 2016 21:46:59 -0000 [ https://issues.apache.org/jira/browse/REEF-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351899#comment-15351899 ] Dhruv Mahajan commented on REEF-1456: ------------------------------------- [~markus.weimer] [~anupam128] So I talked offline to [~chris.douglas] also about possibility of implementing Distributed counters as a special Sink implementation and it seemed possible at higher level. So my plan is to explore the feasibility of implementing equivalent of basic version of metrics2 (heavy borrowing) followed by MapReduce style counters on top of it. > Develop distributed counters framework in REEF > ---------------------------------------------- > > Key: REEF-1456 > URL: https://issues.apache.org/jira/browse/REEF-1456 > Project: REEF > Issue Type: Sub-task > Components: REEF.NET > Environment: C# > Reporter: Dhruv Mahajan > > The aim of this JIRA is to develop distributed counters framework in REEF. Each task can emit pairs of {counter name, incremental value} which are aggregated and sent to driver, which can then aggregate them from all the tasks/ evaluators. The aggregation strategy can be simple addition, most recently used, etc. Via these counters, we can implement some common metrics for ML - like amount (bytes) of data read, current loss function value etc. > We will develop interfaces for Counters for evaluators and driver along with default implementations and aggregation strategies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)