hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sascha Jonas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-681) Multi Layer Perceptron
Date Wed, 15 May 2013 09:25:15 GMT

    [ https://issues.apache.org/jira/browse/HAMA-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13658192#comment-13658192
] 

Sascha Jonas commented on HAMA-681:
-----------------------------------

Hi,

i am writing my master-thesis about distributing a Multilayer Perceptron with Apache Hama
under supervision of Prof. Herta. 

Right now i am implementing batch gradient descent:

- each task calculates the delta Ws (for all connection matrices) for a mini batch given
- the delta Ws are summed up and sended to a master
- the master sums up all delta Ws and calculates the average delta W
- the master calculates the new connection matrices
- the master sends the new connection matrices to all clients
- repeat above steps till number of iterations is reached

Restrictions: 

- the model must fit into memory

TODO:

- implement a train batch method. right now i am using the train online method.
- implement stochastic gradient descent
- replace mahout math libraries with http://mikiobraun.github.io/jblas/ which are 10x faster

You can find the alpha code here:
https://code.google.com/p/large-scale-mlp/

I think it would be great working together on this task. 




                
> Multi Layer Perceptron 
> -----------------------
>
>                 Key: HAMA-681
>                 URL: https://issues.apache.org/jira/browse/HAMA-681
>             Project: Hama
>          Issue Type: New Feature
>          Components: machine learning
>            Reporter: Christian Herta
>
> Implementation of a Multilayer Perceptron (Neural Network)
>  - Learning by Backpropagation 
>  - Distributed Learning
> The implementation should be the basis for the long range goals:
>  - more efficent learning (Adagrad, L-BFGS)
>  - High efficient distributed Learning
>  - Autoencoder - Sparse (denoising) Autoencoder
>  - Deep Learning
>  
> ---
> Due to the overhead of Map-Reduce(MR) MR didn't seem to be the best strategy to distribute
the learning of MLPs.
> Therefore the current implementation of the MLP (see MAHOUT-976) should be migrated to
Hama. First all dependencies to Mahout (Matrix-Library) must be removed to get a standalone
MLP Implementation. Then the Hama BSP programming model should be used to realize distributed
learning.
> Different strategies of efficient synchronized weight updates has to be evaluated.
> Resources:
>  Videos:
>     - http://www.youtube.com/watch?v=ZmNOAtZIgIk
>     - http://techtalks.tv/talks/57639/
>  MLP and Deep Learning Tutorial:
>  - http://www.stanford.edu/class/cs294a/
>  Scientific Papers:
>  - Google's "Brain" project: 
> http://research.google.com/archive/large_deep_networks_nips2012.html
>  - Neural Networks and BSP: http://ipdps.cc.gatech.edu/1998/biosp3/bispp4.pdf
>  - http://jmlr.csail.mit.edu/papers/volume11/vincent10a/vincent10a.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message