hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 姜页希 <yexiji...@gmail.com>
Subject Re: [jira] [Commented] (HAMA-681) Multi Layer Perceptron
Date Wed, 15 May 2013 14:40:51 GMT
Hi, Sascha,

I am also working on this issue. I read your code and it is very nice.

As you mentioned, I found that what your code does is to maintain a copy of
the neural networks in each client tasks, and then each of them conducts
the learning independently. In each iteration, the master task would
synchronize the weight matrix by the messages from the client tasks, and
then it would re-distribute the updated weight matrix to all the client
tasks. This procedure would continue until it meets the maximal iterations.

I also found that your code has dependency on the mahout library, including
the Matrix and Vector.
It is better to reuse the counterparts in package org.apache.hama.ml.math
(or to use the jblas as you mentioned? The link you provide seems invalid.).

I'm wondering whether we can collaborate to do this work? Besides what you
mentioned in the TODO list, we can also do the following things.

1. Make the architecture of the multilayer perceptron more flexible.
There are multiple types of neural networks including feed-forward,
recurrent (good at modeling sequences), and symmetric connected, etc, and
each of them has different rules of connections between neurons.

2. Consider a more scalable mechanism.
This is difficult. Current implementation requires each task to maintain a
copy of the whole architecture. This works perfect for small size
architecture but may not be a good for large size architecture.

3. Add more documentation and test cases.





2013/5/15 Tommaso Teofili (JIRA) <jira@apache.org>

>
>     [
> https://issues.apache.org/jira/browse/HAMA-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13658209#comment-13658209]
>
> Tommaso Teofili commented on HAMA-681:
> --------------------------------------
>
> Hi Sascha, all looking progress here sounds really nice!
>
> [~twist], you could even try to leverage Hama implementation of Gradient
> Descent, see
> -
> http://svn.apache.org/repos/asf/hama/trunk/ml/src/main/java/org/apache/hama/ml/regression/GradientDescentBSP.java
> -
> http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/GradientDescentExample.java
>
> > Multi Layer Perceptron
> > -----------------------
> >
> >                 Key: HAMA-681
> >                 URL: https://issues.apache.org/jira/browse/HAMA-681
> >             Project: Hama
> >          Issue Type: New Feature
> >          Components: machine learning
> >            Reporter: Christian Herta
> >
> > Implementation of a Multilayer Perceptron (Neural Network)
> >  - Learning by Backpropagation
> >  - Distributed Learning
> > The implementation should be the basis for the long range goals:
> >  - more efficent learning (Adagrad, L-BFGS)
> >  - High efficient distributed Learning
> >  - Autoencoder - Sparse (denoising) Autoencoder
> >  - Deep Learning
> >
> > ---
> > Due to the overhead of Map-Reduce(MR) MR didn't seem to be the best
> strategy to distribute the learning of MLPs.
> > Therefore the current implementation of the MLP (see MAHOUT-976) should
> be migrated to Hama. First all dependencies to Mahout (Matrix-Library) must
> be removed to get a standalone MLP Implementation. Then the Hama BSP
> programming model should be used to realize distributed learning.
> > Different strategies of efficient synchronized weight updates has to be
> evaluated.
> > Resources:
> >  Videos:
> >     - http://www.youtube.com/watch?v=ZmNOAtZIgIk
> >     - http://techtalks.tv/talks/57639/
> >  MLP and Deep Learning Tutorial:
> >  - http://www.stanford.edu/class/cs294a/
> >  Scientific Papers:
> >  - Google's "Brain" project:
> > http://research.google.com/archive/large_deep_networks_nips2012.html
> >  - Neural Networks and BSP:
> http://ipdps.cc.gatech.edu/1998/biosp3/bispp4.pdf
> >  - http://jmlr.csail.mit.edu/papers/volume11/vincent10a/vincent10a.pdf
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>



-- 
------
Yexi Jiang,
ECS 251,  yjian004@cs.fiu.edu
School of Computer and Information Science,
Florida International University
Homepage: http://users.cis.fiu.edu/~yjian004/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message