spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mosharaf Chowdhury <mosharafka...@gmail.com>
Subject Re: sync master with slaves with bittorrent?
Date Mon, 19 May 2014 06:22:15 GMT
What twitter calls murder, unless it has changed since then, is just a
BitTornado wrapper. In 2011, We did some comparison on the performance of
murder and the TorrentBroadcast we have right now for Spark's own broadcast
(Section 7.1 in
http://www.mosharaf.com/wp-content/uploads/orchestra-sigcomm11.pdf).
Spark's implementation was 4.5X faster than murder.

The only issue with using TorrentBroadcast to deploy code/VM is writing a
wrapper around it to read from disk, but it shouldn't be too complicated.
If someone picks it up, I can give some pointers on how to proceed (I've
thought about doing it myself forever, but never ended up actually taking
the time; right now I don't have enough free cycles either)

Otherwise, murder/BitTornado would be better than the current strategy we
have.

A third option would be to use rsync; but instead of rsync-ing to every
slave from the master, one can simply rsync from the master first to one
slave; then use the two sources (master and the first slave) to rsync to
two more; then four and so on. Might be a simpler solution without many
changes.

--
Mosharaf Chowdhury
http://www.mosharaf.com/


On Sun, May 18, 2014 at 11:07 PM, Andrew Ash <andrew@andrewash.com> wrote:

> My first thought would be to use libtorrent for this setup, and it turns
> out that both Twitter and Facebook do code deploys with a bittorrent setup.
>  Twitter even released their code as open source:
>
>
> https://blog.twitter.com/2010/murder-fast-datacenter-code-deploys-using-bittorrent
>
>
> http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/
>
>
> On Sun, May 18, 2014 at 10:44 PM, Daniel Mahler <dmahler@gmail.com> wrote:
>
>> I am not an expert in this space either. I thought the initial rsync
>> during launch is really just a straight copy that did not need the tree
>> diff. So it seemed like having the slaves do the copying among it each
>> other would be better than having the master copy to everyone directly.
>> That made me think of bittorrent, though there may well be other systems
>> that do this.
>> From the launches I did today it seems that it is taking around 1 minute
>> per slave to launch a cluster, which can be a problem for clusters with 10s
>> or 100s of slaves, particularly since on ec2  that time has to be paid for.
>>
>>
>> On Sun, May 18, 2014 at 11:54 PM, Aaron Davidson <ilikerps@gmail.com>wrote:
>>
>>> Out of curiosity, do you have a library in mind that would make it easy
>>> to setup a bit torrent network and distribute files in an rsync (i.e.,
>>> apply a diff to a tree, ideally) fashion? I'm not familiar with this space,
>>> but we do want to minimize the complexity of our standard ec2 launch
>>> scripts to reduce the chance of something breaking.
>>>
>>>
>>> On Sun, May 18, 2014 at 9:22 PM, Daniel Mahler <dmahler@gmail.com>wrote:
>>>
>>>> I am launching a rather large cluster on ec2.
>>>> It seems like the launch is taking forever on
>>>> ....
>>>> Setting up spark
>>>> RSYNC'ing /root/spark to slaves...
>>>> ...
>>>>
>>>> It seems that bittorrent might be a faster way to replicate
>>>> the sizeable spark directory to the slaves
>>>> particularly if there is a lot of not very powerful slaves.
>>>>
>>>> Just a thought ...
>>>>
>>>> cheers
>>>> Daniel
>>>>
>>>>
>>>
>>
>

Mime
View raw message