horn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [jira] [Updated] (HORN-8) Implementation of Parameter Server
Date Mon, 09 Nov 2015 04:34:07 GMT
NP, by the way, I'm not sure about it. :-)

According to my co-worker (minho) and Singa slides, Sandblaster L-BFGS
is fit for single node with multi-GPUs and difficult to scale out.

On Mon, Nov 9, 2015 at 1:21 PM, 이기석 <ks881115@gmail.com> wrote:
> Ah, i got it. I think I should read the paper again.
> Thx for the reply :)
>
> 2015-11-09 13:03 GMT+09:00 Edward J. Yoon <edwardyoon@apache.org>:
>
>> Hi Lee,
>>
>> As far as I know, Sandblaster L-BFGS (described in Google's
>> DistBelief) introduces similar optimization mechanism to the Stale
>> Synchronous Parallel, using coordinator.
>>
>> On Mon, Nov 9, 2015 at 12:46 PM, 이기석 <ks881115@gmail.com> wrote:
>> > I have a short question about the Parameter Server access:
>> >
>> > Can't we apply the Stale Synchronous Parallel[2] instead of purely
>> > asynchronous approach? which was introduced as a good compromise between
>> > BSP and asynchronous.
>> >
>> >
>> > [1] Eric P Xing, et al. *Petuum: A New Platform for Distributed Machine
>> > Learning on Big Data*, SIGKDD 2015
>> > paper: http://arxiv.org/pdf/1312.7651v2.pdf
>> > slides: http://petuum.github.io/papers/SysAlgTheoryKDD2015.pdf
>> >
>> > [2] Qirong Ho, et al. *More Effective Distributed ML via a Stale
>> > Synchronous Parallel Parameter Server*, NIPS 2013
>> > paper: http://www.cs.cmu.edu/~epxing/papers/2013/SSPTable_NIPS2013.pdf
>> > slides: http://www.cs.cmu.edu/~qho/ssp_nips2013.pdf
>> >
>> > 2015-11-09 12:27 GMT+09:00 Edward J. Yoon (JIRA) <jira@apache.org>:
>> >
>> >>
>> >>      [
>> >>
>> https://issues.apache.org/jira/browse/HORN-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> >> ]
>> >>
>> >> Edward J. Yoon updated HORN-8:
>> >> ------------------------------
>> >>     Description:
>> >> The current implementation works in synchronous way like below
>> >> (SmallLayeredNeuralNetworkTrainer.java 101 lines):
>> >> {code}
>> >> task0        task1        task2
>> >>
>> >>       compute updates locally
>> >>
>> >> -------------------------------- sends updates to master task
>> >> -------------------------------- merge updates and broadcast it to every
>> >> tasks
>> >>
>> >>       compute updates locally
>> >>
>> >> -------------------------------- sends updates to master task
>> >> -------------------------------- merge updates and broadcast it to every
>> >> tasks
>> >>
>> >>                ...
>> >>
>> >>       (Loop until onvergence)
>> >> {code}
>> >>
>> >> By separating the master, we can support asynchronous parallel SGD. My
>> >> idea is just using of a task0 (BSPTask) as a server daemon. In this
>> issue
>> >> ticket, single master is enough at this moment.
>> >>
>> >> {code}
>> >> task0     |          task1                          ....   taskN
>> >>           |
>> >>           |
>> >>           |   compute updates locally
>> >>           |
>> >>  Receive  |<------ push updates to master task
>> >>  Update1  |
>> >>           +------> fetch updates
>> >>           |
>> >>           |
>> >>           |
>> >>  Receive  |<------------------------------------ ..
>> >>  Update2  |
>> >>           +------------------------------------> ..
>> >>           |
>> >>           |
>> >> {code}
>> >>
>> >>
>> >>   was:
>> >> The current implementation works in synchronous way like below:
>> >> {code}
>> >> task0        task1        task2
>> >>
>> >>       compute updates locally
>> >>
>> >> -------------------------------- sends updates to master task
>> >> -------------------------------- merge updates and broadcast it to every
>> >> tasks
>> >>
>> >>       compute updates locally
>> >>
>> >> -------------------------------- sends updates to master task
>> >> -------------------------------- merge updates and broadcast it to every
>> >> tasks
>> >>
>> >>                ...
>> >>
>> >>       (Loop until onvergence)
>> >> {code}
>> >>
>> >> By separating the master, we can support asynchronous parallel SGD. My
>> >> idea is just using of a task0 (BSPTask) as a server daemon. In this
>> issue
>> >> ticket, single master is enough at this moment.
>> >>
>> >> {code}
>> >> task0     |          task1                          ....   taskN
>> >>           |
>> >>           |
>> >>           |   compute updates locally
>> >>           |
>> >>  Receive  |<------ push updates to master task
>> >>  Update1  |
>> >>           +------> fetch updates
>> >>           |
>> >>           |
>> >>           |
>> >>  Receive  |<------------------------------------ ..
>> >>  Update2  |
>> >>           +------------------------------------> ..
>> >>           |
>> >>           |
>> >> {code}
>> >>
>> >>
>> >>
>> >> > Implementation of Parameter Server
>> >> > ----------------------------------
>> >> >
>> >> >                 Key: HORN-8
>> >> >                 URL: https://issues.apache.org/jira/browse/HORN-8
>> >> >             Project: Apache Horn
>> >> >          Issue Type: Improvement
>> >> >            Reporter: Edward J. Yoon
>> >> >
>> >> > The current implementation works in synchronous way like below
>> >> (SmallLayeredNeuralNetworkTrainer.java 101 lines):
>> >> > {code}
>> >> > task0        task1        task2
>> >> >       compute updates locally
>> >> > -------------------------------- sends updates to master task
>> >> > -------------------------------- merge updates and broadcast it to
>> every
>> >> tasks
>> >> >       compute updates locally
>> >> > -------------------------------- sends updates to master task
>> >> > -------------------------------- merge updates and broadcast it to
>> every
>> >> tasks
>> >> >
>> >> >                ...
>> >> >       (Loop until onvergence)
>> >> > {code}
>> >> > By separating the master, we can support asynchronous parallel SGD.
My
>> >> idea is just using of a task0 (BSPTask) as a server daemon. In this
>> issue
>> >> ticket, single master is enough at this moment.
>> >> > {code}
>> >> > task0     |          task1                          ....   taskN
>> >> >           |
>> >> >           |
>> >> >           |   compute updates locally
>> >> >           |
>> >> >  Receive  |<------ push updates to master task
>> >> >  Update1  |
>> >> >           +------> fetch updates
>> >> >           |
>> >> >           |
>> >> >           |
>> >> >  Receive  |<------------------------------------ ..
>> >> >  Update2  |
>> >> >           +------------------------------------> ..
>> >> >           |
>> >> >           |
>> >> > {code}
>> >>
>> >>
>> >>
>> >> --
>> >> This message was sent by Atlassian JIRA
>> >> (v6.3.4#6332)
>> >>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>>



-- 
Best Regards, Edward J. Yoon

Mime
View raw message