Return-Path: X-Original-To: apmail-singa-dev-archive@minotaur.apache.org Delivered-To: apmail-singa-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2D195182BB for ; Tue, 19 Jan 2016 14:19:43 +0000 (UTC) Received: (qmail 4754 invoked by uid 500); 19 Jan 2016 14:19:43 -0000 Delivered-To: apmail-singa-dev-archive@singa.apache.org Received: (qmail 4726 invoked by uid 500); 19 Jan 2016 14:19:43 -0000 Mailing-List: contact dev-help@singa.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@singa.incubator.apache.org Delivered-To: mailing list dev@singa.incubator.apache.org Received: (qmail 4715 invoked by uid 99); 19 Jan 2016 14:19:43 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2016 14:19:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 95390C30FE for ; Tue, 19 Jan 2016 14:19:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.979 X-Spam-Level: X-Spam-Status: No, score=0.979 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id YMCsOryHVtDm for ; Tue, 19 Jan 2016 14:19:42 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id 062B620CB8 for ; Tue, 19 Jan 2016 14:19:40 +0000 (UTC) Received: (qmail 4464 invoked by uid 99); 19 Jan 2016 14:19:39 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2016 14:19:39 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id C51DE2C14F2 for ; Tue, 19 Jan 2016 14:19:39 +0000 (UTC) Date: Tue, 19 Jan 2016 14:19:39 +0000 (UTC) From: "wangwei (JIRA)" To: dev@singa.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SINGA-130) Implement a layer subclass for data prefetching MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SINGA-130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangwei updated SINGA-130: -------------------------- Assignee: Anh Dinh > Implement a layer subclass for data prefetching > ----------------------------------------------- > > Key: SINGA-130 > URL: https://issues.apache.org/jira/browse/SINGA-130 > Project: Singa > Issue Type: New Feature > Reporter: wangwei > Assignee: Anh Dinh > Labels: data, multi-threading, prefetch > > Data prefetching is important for training with GPU, because the IO would become the bottleneck when the computation is very fast. > One idea is to create a general prefetch layer which embeds the application specific data loading layers. > {code} > PrefetchLayer::ComptueFeature() { > wait until the pretch thread finishes. > swap the prefeth_data_ and data_ blobs. > if (first time) > load data into data_ blobs > spawn a new thread to call functions from data loading layers for loading data into prefetch_data_. > } > {code} > > If the prefetch layer has multiple loading layers and is connected to multiple destination layers, then different destination layer may want data loaded by different loading layers. This case should be handled properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)