singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wang Wei <wang...@comp.nus.edu.sg>
Subject Re: Caffe con Troll and SINGA?
Date Mon, 27 Jul 2015 03:10:48 GMT
Hi Ce,

Thanks for the clarification!
I think I have got the high-level abstractions of CcT.
We will come back to get your help (e.g., transfer some code) after
finishing the GPU version.

Regards,
Wei

On Sat, Jul 25, 2015 at 12:13 PM, Ce Zhang <czhang@cs.wisc.edu> wrote:

> >> How do you parallelize the training on CPU and GPU, by creating two
> threads each with one (Caffe) Solver?
> >> Or does each Layer partition the mini-batch and dispatch them onto two
> threads (one using GPU Driver and one using CPU >> Driver) after receiving
> the Pointer?
>
> The abstraction is like this. A Layer is an abstraction that takes as
> input a Pointer and fills an output Pointer. This allows us to have
> two types of Layers:
>    1. Single-Device Layer, which corresponding to one worker that only
> works on a single device.
>    2. Multi-Device Layer, which creates multiple single-device layers,
> starts multiple threads
> correspond to these single-device layers, call their forward/backward
> function
> with p_thread, and join these threads.
>
> Both types of Layers are the subclass of the same Layer class so they share
> the same interface.
>
> With this abstraction, I think this corresponds to your second strategy.
>
> >> Do I miss any other features of CcT?
>
> I think these 4-steps are exactly right, and this should be enough
> to integrate all features of CcT!
>
> One very minor thing to notice is that the parallelization we talked about
> maintains the replica of models on different devices and partition the
> data.
> It is also possible to partition the model instead, which might work
> better
> for layers like FC--I think a similar tradeoff is also in SINGA's
> technical
> report, so I am pretty sure this case will be handled efficiently! :)
>
> Thanks again for integrating CcT into SINGA! Let us know if you have any
> questions!
>
> Ce
>
> On Fri, Jul 24, 2015 at 10:40 PM, Wang Wei <wangwei@comp.nus.edu.sg>
> wrote:
>
>> Hi Ce,
>>
>> Thanks for your explanation.
>> I read the example training configuration of CcT on github, and have some
>> further questions on the parallelization.
>>
>> How do you parallelize the training on CPU and GPU, by creating two
>> threads each with one (Caffe) Solver?
>> Or does each Layer partition the mini-batch and dispatch them onto two
>> threads (one using GPU Driver and one using CPU Driver) after receiving the
>> Pointer?
>>
>> According to the configuration file (
>> https://github.com/nudles/CaffeConTroll/blob/master/tests/imagenet_train/train_val/alexnet_train_val_1GPU_CPU.prototxt),
>>
>> it seems the parallelization is at the Layer level (i.e, the second one).
>> In specific, given two connected layers A->B (e.g., conv and relu).
>> If the batch partitioning for A is 0.6 on GPU and 0.4 on CPU, while the
>> partitioning for B is 1.0 (ie.. all) on GPU.
>> Then the computation for B should be blocked until the computation for A
>> is finished.
>> Consequently, the synchronization is done for every layer.
>>
>> For SINGA, we use the worker-server architecture (similar to parameter
>> server).
>> Currently, we support both synchronous and asynchronous training on CPU.
>> We call the training framework implemented by CcT as synchronous training.
>> SINGA implements this training framework by partitioning the neural
>> network among one worker group.  One worker runs in a thread.
>> The partitioning is currently done at Layer level. Users can configure it
>> to be on dimension 0  or 1 of the feature blob.
>> For dimension 0, it means partitioning one mini-batch onto workers like
>> CcT (but we use equal partitioning).
>> For dimension 1, it means partitioning one layer (with 4096 neurons) into
>> sub-layers (with 1024 neurons if there are 4 workers).
>> After partitioning, each (sub) layer will be assigned a location ID,
>> i.e., the ID of the worker to which the (sub) layer will be dispatched.
>> We will support partitioning at the neural network level, i.e., let users
>> to configure the location ID of one layer.
>> During training, each worker it has the full neural network structure,
>> but it only visits (e.g., forward pass) the layers that are dispatched to
>> him (based on Location ID and worker ID).
>>
>>
>> To Support CcT in SINGA,
>> 1. We first need to support GPU training (should be done in August).
>> 2. Update the neural network partitioning function to integrate CcT's
>> scheduling strategy.
>> 2. Make the Worker class a template, and create GPU workers and CPU
>> workers after partitioning the neural network.
>> 3. I think the Lowering, Shifting techniques are easy to integrate as a
>> library if it is independent of the devices.
>>
>> Do I miss any other features of CcT?
>>
>> Regards,
>> Wei
>>
>> A Layer takes as input a Pointer, and calls the `dereference` function of
>>> Pointer to
>>> obtain local-copy (w.r.t. the driver) of the data. Therefore, the Layer
>>> object does not
>>> know where the input data comes from.
>>>
>>> To run operations like GEMM, Lowering, a Layer will call Driver, which
>>> provided
>>> a unified interface across devices.
>>>
>>> The Layer, Pointer and Driver abstractions are clear and easy to
>> understand.
>>
>>> I think it is possible for you to compile each layer as a library that
>>> you can call that
>>> takes as input a pointer object and fills in another pointer object as
>>> output.
>>>
>>> >> How do you synchronize the parameters trained on CPU and GPU, using
>>> the implementation of Hogwild from Caffe?
>>>
>>> Currently, we parallelize inside a batch--If the batch size is 256, we
>>> might put 200 of them on GPU, 56 of
>>> them on CPU. After their result comes back, we aggregate the result.
>>> This means that our current result
>>> is exactly the same as a single-thread run.
>>>
>>>
>>
>>
>>> For AlexNet with 256 batch size (the one most paper used), we observe
>>> this strategy gives almost
>>> linear speed up even with 4 Amazon EC2 GPUs.
>>>
>>> Of course Hogwild! or parameter servers are a natural direction to
>>> further scale up the current system
>>> when the number of computational devices further increase and the
>>> aggregation time starts to
>>> dominate...
>>>
>>> >> Which part of Caffe you have changed (we also borrow some code from
>>> Caffe, thus know its structure)?
>>>
>>> We borrow the parser code, loader code, and protobuf code. The main
>>> reason is to make sure CcT
>>> is compatible with Caffe. I think we rewrite most other layers,
>>> especially for CONV. For faster layers
>>> like ReLU, our code are very similar to them.
>>>
>>> Let us know if you have any questions!
>>>
>>> Ce
>>>
>>> On Wed, Jul 22, 2015 at 10:49 PM, Wang Wei <wangwei@comp.nus.edu.sg>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Jul 23, 2015 at 11:47 AM, Wang Wei <wangwei@comp.nus.edu.sg>
>>>> wrote:
>>>>
>>>>> Hi Ce,
>>>>>
>>>>> Thanks for starting the discussion.
>>>>>
>>>>> We are preparing documentations and test for our first Apache
>>>>> Incubator release.
>>>>> I planed to contact Stefan after the first release. Because the first
>>>>> release does not support GPU.
>>>>> There are some developers in NetEase working on GPU implementation,
>>>>> which will be integrated for the second release.
>>>>>
>>>>> It is a good time to discuss the integration now. We can consider this
>>>>> new feature while implementing the GPU version.
>>>>> Currently, we use Blob (from caffe) to manage memory and Mshadow for
>>>>> computation.
>>>>> To integerate Caffe on Troll, the ideal case is making Cafee on Troll
>>>>> a library like Mshadow.
>>>>> I think at least the convolution optimization techniques (lowering,
>>>>> multiply, lifting) could be compiled as a library (correct?).
>>>>>
>>>>> How do you manage the memory across CPU and GPU?
>>>>> How do you synchronize the parameters trained on CPU and GPU, using
>>>>> the implementation of Hogwild from Caffe?
>>>>> Which part of Caffe you have changed (we also borrow some code from
>>>>> Caffe, thus know its structure)?
>>>>>
>>>>> Thank you.
>>>>> (I cc'ed our dev-mailing list to notify the developers on GPU)
>>>>>
>>>>> Regards,
>>>>> Wei Wang
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 23, 2015 at 9:47 AM, Ce Zhang <czhang@cs.wisc.edu>
wrote:
>>>>>
>>>>>> Hi Wei,
>>>>>>
>>>>>> I am Ce from Wisconsin and Chris Re's group.
>>>>>> I am one of the developer of Caffe con Troll
>>>>>> (the CNN system that is faster than Caffe on
>>>>>> CPU and can run hybrid between CPU and GPU)
>>>>>>
>>>>>> I think you and Stefan (CC'ed) chat at SIGMOD
>>>>>> about the possibility of integrating Caffe con Troll into
>>>>>> Apache SINGA. We are very excited about this!
>>>>>>
>>>>>> We are very curious about how to make this happen,
>>>>>> e.g., what information do you need from us to do
>>>>>> such an integration. This email aims at starting this
>>>>>> discussion, and we'd love to hear your opinions.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Ce
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message