spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mosharaf Chowdhury <>
Subject Re: sync master with slaves with bittorrent?
Date Mon, 19 May 2014 06:22:15 GMT
What twitter calls murder, unless it has changed since then, is just a
BitTornado wrapper. In 2011, We did some comparison on the performance of
murder and the TorrentBroadcast we have right now for Spark's own broadcast
(Section 7.1 in
Spark's implementation was 4.5X faster than murder.

The only issue with using TorrentBroadcast to deploy code/VM is writing a
wrapper around it to read from disk, but it shouldn't be too complicated.
If someone picks it up, I can give some pointers on how to proceed (I've
thought about doing it myself forever, but never ended up actually taking
the time; right now I don't have enough free cycles either)

Otherwise, murder/BitTornado would be better than the current strategy we

A third option would be to use rsync; but instead of rsync-ing to every
slave from the master, one can simply rsync from the master first to one
slave; then use the two sources (master and the first slave) to rsync to
two more; then four and so on. Might be a simpler solution without many

Mosharaf Chowdhury

On Sun, May 18, 2014 at 11:07 PM, Andrew Ash <> wrote:

> My first thought would be to use libtorrent for this setup, and it turns
> out that both Twitter and Facebook do code deploys with a bittorrent setup.
>  Twitter even released their code as open source:
> On Sun, May 18, 2014 at 10:44 PM, Daniel Mahler <> wrote:
>> I am not an expert in this space either. I thought the initial rsync
>> during launch is really just a straight copy that did not need the tree
>> diff. So it seemed like having the slaves do the copying among it each
>> other would be better than having the master copy to everyone directly.
>> That made me think of bittorrent, though there may well be other systems
>> that do this.
>> From the launches I did today it seems that it is taking around 1 minute
>> per slave to launch a cluster, which can be a problem for clusters with 10s
>> or 100s of slaves, particularly since on ec2  that time has to be paid for.
>> On Sun, May 18, 2014 at 11:54 PM, Aaron Davidson <>wrote:
>>> Out of curiosity, do you have a library in mind that would make it easy
>>> to setup a bit torrent network and distribute files in an rsync (i.e.,
>>> apply a diff to a tree, ideally) fashion? I'm not familiar with this space,
>>> but we do want to minimize the complexity of our standard ec2 launch
>>> scripts to reduce the chance of something breaking.
>>> On Sun, May 18, 2014 at 9:22 PM, Daniel Mahler <>wrote:
>>>> I am launching a rather large cluster on ec2.
>>>> It seems like the launch is taking forever on
>>>> ....
>>>> Setting up spark
>>>> RSYNC'ing /root/spark to slaves...
>>>> ...
>>>> It seems that bittorrent might be a faster way to replicate
>>>> the sizeable spark directory to the slaves
>>>> particularly if there is a lot of not very powerful slaves.
>>>> Just a thought ...
>>>> cheers
>>>> Daniel

View raw message