singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangwei (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SINGA-130) Implement a layer subclass for data prefetching
Date Mon, 11 Jan 2016 14:48:40 GMT
wangwei created SINGA-130:
-----------------------------

             Summary: Implement a layer subclass for data prefetching
                 Key: SINGA-130
                 URL: https://issues.apache.org/jira/browse/SINGA-130
             Project: Singa
          Issue Type: New Feature
            Reporter: wangwei


Data prefetching is important for training with GPU, because the IO would become the bottleneck
when the computation is very fast.

One idea is to create a general prefetch layer which embeds the application specific data
loading layers. 

{code}
PrefetchLayer::ComptueFeature() {
  wait until the pretch thread finishes.
  swap the prefeth_data_ and data_ blobs.
  if (first time)
     load data into data_ blobs
  spawn a new thread to call functions from data loading layers for loading data into prefetch_data_.
}
{code}
 
If the prefetch layer has multiple loading layers and is connected to multiple destination
layers, then different destination layer may want data loaded by different loading layers.
This case should be handled properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message