horn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Mehta <shubham.meht...@gmail.com>
Subject Re: Parallelizing SGD for CNN model
Date Wed, 14 Oct 2015 03:51:20 GMT
Hi Edward,

If I am not wrong you mean techiniques like this...
*Fully Convolutional Networks for Semantic Segmentation*
http://www.cs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf

Yes, you are right that it has 2 times layers and parameters. But in
distributed framework where we are handling computation at neuron level
memory will not be a issue. These kind of models have large parameters in
total but if we break them by layer then parameters that one layer has to
handle are similar to CNN.

On Tue, Oct 13, 2015 at 7:56 PM, Edward J. Yoon <edwardyoon@apache.org>
wrote:

> You're right. I mean, there're some techniques that using conv-deconv
> together for semantic segmentation. So, 2 times layers and parameters
> are needed. I heard that it was difficult to learn by memory
> limitation from them.
>
> On Tue, Oct 13, 2015 at 7:52 PM, Shubham Mehta
> <shubham.mehta93@gmail.com> wrote:
> > Hi,
> >
> > As far as I understood De-convolutional NN (DNN) also have same number of
> > parameters as Convolutional NN (CNN). The only difference between two is
> > Deconvolutional uses filters for deconvoluting image and generated
> feature
> > map is transferred to next layer while Convolutional uses filters to
> > convolute with image and resultant is transferred to next layer.
> >
> > Even in DNN ( CNN ) computation is much more in deconvolution layers (
> > convolutional layers ) as compared to fully connected layers and
> parameters
> > are much lesser. So the architecture that we use for DNN will be similar
> to
> > what we use for CNN. Only the operation at each Neuron/Layer changes.
> >
> > Please correct me if I'm wrong in my understanding.
> >
> > Regards,
> > Shubham
> >
> >
> >
> > On Tue, Oct 13, 2015 at 7:22 PM, Edward J. Yoon <edwardyoon@apache.org>
> > wrote:
> >
> >> Hi,
> >>
> >> There's also a de-convolutional networks that requires huge memory for
> >> large number of parameters.
> >>
> >> On Tue, Oct 13, 2015 at 6:46 PM, Shubham Mehta
> >> <shubham.mehta93@gmail.com> wrote:
> >> > Hello, everyone
> >> >
> >> > A very good read on parallelizing Convolutional Neural Network.
> >> >
> >> > http://arxiv.org/abs/1404.5997
> >> >
> >> > Apache Singa follows this approach specifically for CNN.
> >> >
> >> > The gist of paper is that data parallelism is more beneficial for
> >> > Convolutional layers while model parallelism for fully-connected
> layers.
> >> >
> >> >
> >> > Regards
> >> > Shubam Mehta
> >> >
> >> > --
> >> > Shubham Mehta
> >> > B.Tech 2015
> >> > Computer Science and Engineering
> >> > IIT Bombay
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >>
> >
> >
> >
> > --
> > Shubham Mehta
> > B.Tech 2015
> > Computer Science and Engineering
> > IIT Bombay
>
>
>
> --
> Best Regards, Edward J. Yoon
>



-- 
Shubham Mehta
B.Tech 2015
Computer Science and Engineering
IIT Bombay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message