singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (SINGA-130) Implement a layer subclass for data prefetching
Date Fri, 08 Apr 2016 02:39:25 GMT


ASF subversion and git services commented on SINGA-130:

Commit 4344ab7b2a17711d77465a6c52e50b41cad0dfef in incubator-singa's branch refs/heads/master
from [~ug93tad]
[;h=4344ab7 ]

SINGA-130 Data prefetching

Move fetch_data thread to ComputeFeature()

Pass cpplint check

> Implement a layer subclass for data prefetching
> -----------------------------------------------
>                 Key: SINGA-130
>                 URL:
>             Project: Singa
>          Issue Type: New Feature
>            Reporter: wangwei
>            Assignee: Anh Dinh
>              Labels: data, multi-threading, prefetch
> Data prefetching is important for training with GPU, because the IO would become the
bottleneck when the computation is very fast.
> One idea is to create a general prefetch layer which embeds the application specific
data loading layers. 
> {code}
> PrefetchLayer::ComptueFeature() {
>   wait until the pretch thread finishes.
>   swap the prefeth_data_ and data_ blobs.
>   if (first time)
>      load data into data_ blobs
>   spawn a new thread to call functions from data loading layers for loading data into
> }
> {code}
> If the prefetch layer has multiple loading layers and is connected to multiple destination
layers, then different destination layer may want data loaded by different loading layers.
This case should be handled properly.

This message was sent by Atlassian JIRA

View raw message