horn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 이기석 <ks881...@gmail.com>
Subject Re: [jira] [Updated] (HORN-8) Implementation of Parameter Server
Date Mon, 09 Nov 2015 04:21:21 GMT
Ah, i got it. I think I should read the paper again.
Thx for the reply :)

2015-11-09 13:03 GMT+09:00 Edward J. Yoon <edwardyoon@apache.org>:

> Hi Lee,
>
> As far as I know, Sandblaster L-BFGS (described in Google's
> DistBelief) introduces similar optimization mechanism to the Stale
> Synchronous Parallel, using coordinator.
>
> On Mon, Nov 9, 2015 at 12:46 PM, 이기석 <ks881115@gmail.com> wrote:
> > I have a short question about the Parameter Server access:
> >
> > Can't we apply the Stale Synchronous Parallel[2] instead of purely
> > asynchronous approach? which was introduced as a good compromise between
> > BSP and asynchronous.
> >
> >
> > [1] Eric P Xing, et al. *Petuum: A New Platform for Distributed Machine
> > Learning on Big Data*, SIGKDD 2015
> > paper: http://arxiv.org/pdf/1312.7651v2.pdf
> > slides: http://petuum.github.io/papers/SysAlgTheoryKDD2015.pdf
> >
> > [2] Qirong Ho, et al. *More Effective Distributed ML via a Stale
> > Synchronous Parallel Parameter Server*, NIPS 2013
> > paper: http://www.cs.cmu.edu/~epxing/papers/2013/SSPTable_NIPS2013.pdf
> > slides: http://www.cs.cmu.edu/~qho/ssp_nips2013.pdf
> >
> > 2015-11-09 12:27 GMT+09:00 Edward J. Yoon (JIRA) <jira@apache.org>:
> >
> >>
> >>      [
> >>
> https://issues.apache.org/jira/browse/HORN-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> >> ]
> >>
> >> Edward J. Yoon updated HORN-8:
> >> ------------------------------
> >>     Description:
> >> The current implementation works in synchronous way like below
> >> (SmallLayeredNeuralNetworkTrainer.java 101 lines):
> >> {code}
> >> task0        task1        task2
> >>
> >>       compute updates locally
> >>
> >> -------------------------------- sends updates to master task
> >> -------------------------------- merge updates and broadcast it to every
> >> tasks
> >>
> >>       compute updates locally
> >>
> >> -------------------------------- sends updates to master task
> >> -------------------------------- merge updates and broadcast it to every
> >> tasks
> >>
> >>                ...
> >>
> >>       (Loop until onvergence)
> >> {code}
> >>
> >> By separating the master, we can support asynchronous parallel SGD. My
> >> idea is just using of a task0 (BSPTask) as a server daemon. In this
> issue
> >> ticket, single master is enough at this moment.
> >>
> >> {code}
> >> task0     |          task1                          ....   taskN
> >>           |
> >>           |
> >>           |   compute updates locally
> >>           |
> >>  Receive  |<------ push updates to master task
> >>  Update1  |
> >>           +------> fetch updates
> >>           |
> >>           |
> >>           |
> >>  Receive  |<------------------------------------ ..
> >>  Update2  |
> >>           +------------------------------------> ..
> >>           |
> >>           |
> >> {code}
> >>
> >>
> >>   was:
> >> The current implementation works in synchronous way like below:
> >> {code}
> >> task0        task1        task2
> >>
> >>       compute updates locally
> >>
> >> -------------------------------- sends updates to master task
> >> -------------------------------- merge updates and broadcast it to every
> >> tasks
> >>
> >>       compute updates locally
> >>
> >> -------------------------------- sends updates to master task
> >> -------------------------------- merge updates and broadcast it to every
> >> tasks
> >>
> >>                ...
> >>
> >>       (Loop until onvergence)
> >> {code}
> >>
> >> By separating the master, we can support asynchronous parallel SGD. My
> >> idea is just using of a task0 (BSPTask) as a server daemon. In this
> issue
> >> ticket, single master is enough at this moment.
> >>
> >> {code}
> >> task0     |          task1                          ....   taskN
> >>           |
> >>           |
> >>           |   compute updates locally
> >>           |
> >>  Receive  |<------ push updates to master task
> >>  Update1  |
> >>           +------> fetch updates
> >>           |
> >>           |
> >>           |
> >>  Receive  |<------------------------------------ ..
> >>  Update2  |
> >>           +------------------------------------> ..
> >>           |
> >>           |
> >> {code}
> >>
> >>
> >>
> >> > Implementation of Parameter Server
> >> > ----------------------------------
> >> >
> >> >                 Key: HORN-8
> >> >                 URL: https://issues.apache.org/jira/browse/HORN-8
> >> >             Project: Apache Horn
> >> >          Issue Type: Improvement
> >> >            Reporter: Edward J. Yoon
> >> >
> >> > The current implementation works in synchronous way like below
> >> (SmallLayeredNeuralNetworkTrainer.java 101 lines):
> >> > {code}
> >> > task0        task1        task2
> >> >       compute updates locally
> >> > -------------------------------- sends updates to master task
> >> > -------------------------------- merge updates and broadcast it to
> every
> >> tasks
> >> >       compute updates locally
> >> > -------------------------------- sends updates to master task
> >> > -------------------------------- merge updates and broadcast it to
> every
> >> tasks
> >> >
> >> >                ...
> >> >       (Loop until onvergence)
> >> > {code}
> >> > By separating the master, we can support asynchronous parallel SGD. My
> >> idea is just using of a task0 (BSPTask) as a server daemon. In this
> issue
> >> ticket, single master is enough at this moment.
> >> > {code}
> >> > task0     |          task1                          ....   taskN
> >> >           |
> >> >           |
> >> >           |   compute updates locally
> >> >           |
> >> >  Receive  |<------ push updates to master task
> >> >  Update1  |
> >> >           +------> fetch updates
> >> >           |
> >> >           |
> >> >           |
> >> >  Receive  |<------------------------------------ ..
> >> >  Update2  |
> >> >           +------------------------------------> ..
> >> >           |
> >> >           |
> >> > {code}
> >>
> >>
> >>
> >> --
> >> This message was sent by Atlassian JIRA
> >> (v6.3.4#6332)
> >>
>
>
>
> --
> Best Regards, Edward J. Yoon
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message