horn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HORN-4) Designing Programming APIs
Date Mon, 26 Oct 2015 07:37:27 GMT

    [ https://issues.apache.org/jira/browse/HORN-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973582#comment-14973582
] 

Edward J. Yoon edited comment on HORN-4 at 10/26/15 7:37 AM:
-------------------------------------------------------------

Here's my pseudo code for basic neural network based on neuron-centric programming model.
I guess this programming model can cover all neural nets in flexible way. 

{code}
 class MyNeuron extends Neuron
   method upward(messages [m1, m2, ..., ])
       sum ← 0
       for each w ∈ [m1, m2, ..., ] do
          sum  ← sum + m.input * m.weight
          
       // propagate squashed output value to neurons of next layer
       propagate(squashingFunction(sum));

   method downward(messages [m1, m2, ..., ])
       for each w ∈ [m1, m2, ..., ] do
         gradient  ← this.output * (1 - this.output) * m.delta *  m.weight
         propagate(gradient);

         // weight collections
         w ← w + Δw (α * this.output * m.delta)

       // push updates to parameter server
       push(weights);
{code}

The reason I separate it into two methods (unlike Google's Pregel) is that framework can determine
whether it's upward or downward phase by message type internally. Moreover, with this, we
can reduce the user-side code complexity. The advantage of this is very fit for multi-thread
programming and parallel computing of each gradient calculations at neuron level. Now, I'm
looking at rootbeer for leveraging GPU also. If GPU accelerating is possible with this, I
think this programming model is perfect.


was (Author: udanax):
Here's my pseudo code for basic neural network based on neuron-centric programming model.
I roughly guess this programming model can cover all neural nets in flexible way. Only one
thing is not clear, this neurons (objects) can be ran on GPU efficiently.

{code}
 class MyNeuron extends Neuron
   method upward(messages [m1, m2, ..., ])
       sum ← 0
       for each w ∈ [m1, m2, ..., ] do
          sum  ← sum + m.input * m.weight
          
       // propagate squashed output value to neurons of next layer
       propagate(squashingFunction(sum));

   method downward(messages [m1, m2, ..., ])
       for each w ∈ [m1, m2, ..., ] do
         gradient  ← this.output * (1 - this.output) * m.delta *  m.weight
         propagate(gradient);

         // weight collections
         w ← w + Δw (α * this.output * m.delta)

       // push updates to parameter server
       push(weights);
{code}

> Designing Programming APIs
> --------------------------
>
>                 Key: HORN-4
>                 URL: https://issues.apache.org/jira/browse/HORN-4
>             Project: Apache Horn
>          Issue Type: Sub-task
>          Components: api
>            Reporter: Edward J. Yoon
>
> As described in original proposal, we will have neuron-centric programming interface
(or we can support multiple programming paradigms e.g., layer-centric). Regarding languages,
we're going to be supporting a number of languages such as Java, Python, Scala.
> At the moment, I'll focus only on Java-based neuron-centric in this ticket. 
> Below is my rough sketch. I suggest that we provides two methods: upward and downward.
The reason of separating into two methods is to provide more intuitive programming model and
reduce user-side code complexity. The up or downward can be determined by message type internally,
and basically each neuron can be activated only when the messages are delivered. 
> {code}
> public interface NeuronInterface<M extends Writable> {
>   /**
>    * This method is called when the messages are propagated from the lower
>    * layer. It can be used to determine if the neuron would activate, or fire.
>    * 
>    * @param messages
>    * @throws IOException
>    */
>   public void upward(Iterable<M> messages) throws IOException;
>   /**
>    * This method is called when the errors are propagated from the upper layer.
>    * It can be used to calculate the error of each neuron and change the
>    * weights.
>    * 
>    * @param messages
>    * @throws IOException
>    */
>   public void downward(Iterable<M> messages) throws IOException;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message