mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dick Davies <>
Subject Re: Do i really need HDFS?
Date Tue, 21 Oct 2014 08:57:03 GMT
I think Spark needs a way to send jobs to/from the workers - the Spark
distro itself
will pull down the executor ok, but in my (very basic) tests I got
stuck without HDFS.

So basically it depends on the framework. I think in Sparks case they
assume most
users are migrating from an existing Hadoop deployment, so HDFS is
sort of assumed.

On 20 October 2014 23:18, CCAAT <> wrote:
> On 10/20/14 11:46, Steven Schlansker wrote:
>> We are running Mesos entirely without HDFS with no problems.  We use
>> Docker to distribute our
>> application to slave nodes, and keep no state on individual nodes.
> Background: I'm building up a 3 node cluster to run mesos and spark. No
> legacy Hadoop needed or wanted. I am using btrfs for the local file system,
> with (2) drives set up for raid1 on each system.
> So you  are suggesting that I can install mesos + spark + docker
> and not a DFS on these (3) machines?
> Will I need any other softwares? My application is a geophysical
> fluid simulator, so scala, R, and all sorts of advanced math will
> be required on the cluster for the Finite Element Methods.
> James

View raw message