horn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [jira] [Updated] (HORN-8) Implementation of Parameter Server
Date Mon, 09 Nov 2015 04:03:34 GMT
Hi Lee,

As far as I know, Sandblaster L-BFGS (described in Google's
DistBelief) introduces similar optimization mechanism to the Stale
Synchronous Parallel, using coordinator.

On Mon, Nov 9, 2015 at 12:46 PM, 이기석 <ks881115@gmail.com> wrote:
> I have a short question about the Parameter Server access:
>
> Can't we apply the Stale Synchronous Parallel[2] instead of purely
> asynchronous approach? which was introduced as a good compromise between
> BSP and asynchronous.
>
>
> [1] Eric P Xing, et al. *Petuum: A New Platform for Distributed Machine
> Learning on Big Data*, SIGKDD 2015
> paper: http://arxiv.org/pdf/1312.7651v2.pdf
> slides: http://petuum.github.io/papers/SysAlgTheoryKDD2015.pdf
>
> [2] Qirong Ho, et al. *More Effective Distributed ML via a Stale
> Synchronous Parallel Parameter Server*, NIPS 2013
> paper: http://www.cs.cmu.edu/~epxing/papers/2013/SSPTable_NIPS2013.pdf
> slides: http://www.cs.cmu.edu/~qho/ssp_nips2013.pdf
>
> 2015-11-09 12:27 GMT+09:00 Edward J. Yoon (JIRA) <jira@apache.org>:
>
>>
>>      [
>> https://issues.apache.org/jira/browse/HORN-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Edward J. Yoon updated HORN-8:
>> ------------------------------
>>     Description:
>> The current implementation works in synchronous way like below
>> (SmallLayeredNeuralNetworkTrainer.java 101 lines):
>> {code}
>> task0        task1        task2
>>
>>       compute updates locally
>>
>> -------------------------------- sends updates to master task
>> -------------------------------- merge updates and broadcast it to every
>> tasks
>>
>>       compute updates locally
>>
>> -------------------------------- sends updates to master task
>> -------------------------------- merge updates and broadcast it to every
>> tasks
>>
>>                ...
>>
>>       (Loop until onvergence)
>> {code}
>>
>> By separating the master, we can support asynchronous parallel SGD. My
>> idea is just using of a task0 (BSPTask) as a server daemon. In this issue
>> ticket, single master is enough at this moment.
>>
>> {code}
>> task0     |          task1                          ....   taskN
>>           |
>>           |
>>           |   compute updates locally
>>           |
>>  Receive  |<------ push updates to master task
>>  Update1  |
>>           +------> fetch updates
>>           |
>>           |
>>           |
>>  Receive  |<------------------------------------ ..
>>  Update2  |
>>           +------------------------------------> ..
>>           |
>>           |
>> {code}
>>
>>
>>   was:
>> The current implementation works in synchronous way like below:
>> {code}
>> task0        task1        task2
>>
>>       compute updates locally
>>
>> -------------------------------- sends updates to master task
>> -------------------------------- merge updates and broadcast it to every
>> tasks
>>
>>       compute updates locally
>>
>> -------------------------------- sends updates to master task
>> -------------------------------- merge updates and broadcast it to every
>> tasks
>>
>>                ...
>>
>>       (Loop until onvergence)
>> {code}
>>
>> By separating the master, we can support asynchronous parallel SGD. My
>> idea is just using of a task0 (BSPTask) as a server daemon. In this issue
>> ticket, single master is enough at this moment.
>>
>> {code}
>> task0     |          task1                          ....   taskN
>>           |
>>           |
>>           |   compute updates locally
>>           |
>>  Receive  |<------ push updates to master task
>>  Update1  |
>>           +------> fetch updates
>>           |
>>           |
>>           |
>>  Receive  |<------------------------------------ ..
>>  Update2  |
>>           +------------------------------------> ..
>>           |
>>           |
>> {code}
>>
>>
>>
>> > Implementation of Parameter Server
>> > ----------------------------------
>> >
>> >                 Key: HORN-8
>> >                 URL: https://issues.apache.org/jira/browse/HORN-8
>> >             Project: Apache Horn
>> >          Issue Type: Improvement
>> >            Reporter: Edward J. Yoon
>> >
>> > The current implementation works in synchronous way like below
>> (SmallLayeredNeuralNetworkTrainer.java 101 lines):
>> > {code}
>> > task0        task1        task2
>> >       compute updates locally
>> > -------------------------------- sends updates to master task
>> > -------------------------------- merge updates and broadcast it to every
>> tasks
>> >       compute updates locally
>> > -------------------------------- sends updates to master task
>> > -------------------------------- merge updates and broadcast it to every
>> tasks
>> >
>> >                ...
>> >       (Loop until onvergence)
>> > {code}
>> > By separating the master, we can support asynchronous parallel SGD. My
>> idea is just using of a task0 (BSPTask) as a server daemon. In this issue
>> ticket, single master is enough at this moment.
>> > {code}
>> > task0     |          task1                          ....   taskN
>> >           |
>> >           |
>> >           |   compute updates locally
>> >           |
>> >  Receive  |<------ push updates to master task
>> >  Update1  |
>> >           +------> fetch updates
>> >           |
>> >           |
>> >           |
>> >  Receive  |<------------------------------------ ..
>> >  Update2  |
>> >           +------------------------------------> ..
>> >           |
>> >           |
>> > {code}
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>



-- 
Best Regards, Edward J. Yoon

Mime
View raw message