Return-Path: X-Original-To: apmail-horn-dev-archive@minotaur.apache.org Delivered-To: apmail-horn-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6A8E0180E1 for ; Thu, 19 Nov 2015 02:15:19 +0000 (UTC) Received: (qmail 11493 invoked by uid 500); 19 Nov 2015 02:15:19 -0000 Delivered-To: apmail-horn-dev-archive@horn.apache.org Received: (qmail 11449 invoked by uid 500); 19 Nov 2015 02:15:19 -0000 Mailing-List: contact dev-help@horn.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@horn.incubator.apache.org Delivered-To: mailing list dev@horn.incubator.apache.org Received: (qmail 11438 invoked by uid 99); 19 Nov 2015 02:15:19 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Nov 2015 02:15:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id E5BB31A2DD5 for ; Thu, 19 Nov 2015 02:15:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.971 X-Spam-Level: X-Spam-Status: No, score=0.971 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id IgFu4YgDg985 for ; Thu, 19 Nov 2015 02:15:11 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with SMTP id 81B1D2022D for ; Thu, 19 Nov 2015 02:15:11 +0000 (UTC) Received: (qmail 11169 invoked by uid 99); 19 Nov 2015 02:15:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Nov 2015 02:15:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 072B82C0453 for ; Thu, 19 Nov 2015 02:15:11 +0000 (UTC) Date: Thu, 19 Nov 2015 02:15:11 +0000 (UTC) From: "Edward J. Yoon (JIRA)" To: dev@horn.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HORN-2) Umbrella ticket for Implementation Planning of Apache Horn MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HORN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012634#comment-15012634 ] Edward J. Yoon commented on HORN-2: ----------------------------------- Here's my idea after thinking more: We provides two batch modes for training model: 1) mini-batch 2) online. As you know, the legacy mini-batch SGD code is fit for tensor/matrix approach and GPU can be easily used. Another one is online SGD based on iterative computing like Pregel. It can be slower than mini-batch but can be useful for large model I think. > Umbrella ticket for Implementation Planning of Apache Horn > ---------------------------------------------------------- > > Key: HORN-2 > URL: https://issues.apache.org/jira/browse/HORN-2 > Project: Apache Horn > Issue Type: Wish > Reporter: Edward J. Yoon > > My old rough idea is described here: http://blog.udanax.org/2015/06/googles-distbelief-clone-project-on.html > The basic idea of data and model parallelism is use of the remote parameter server to parallelize model creation and distribute training across machines, and the region barrier synchronization per task group instead of global barrier synchronization for performing asynchronous mini-batches within single BSP job. > Since Apache Hama provides pluggable interface for Synchronization[1], we can easily create our own region barrier synchronization service for handling multiple BSP worker groups (Regarding management of Tasks Topology, I have no concrete idea yet). > Parameter Server requires decision whether to use Legacy open source or implement ourself. > My rough Programming Inteface Design is only focused on feed-forward networks such as MLP, CNN, and Autoencoder. We may want to conver everything. > 1. http://wiki.apache.org/hama/SyncService -- This message was sent by Atlassian JIRA (v6.3.4#6332)