mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dick Davies <d...@hellooperator.net>
Subject Re: Do i really need HDFS?
Date Tue, 21 Oct 2014 08:57:03 GMT
I think Spark needs a way to send jobs to/from the workers - the Spark
distro itself
will pull down the executor ok, but in my (very basic) tests I got
stuck without HDFS.

So basically it depends on the framework. I think in Sparks case they
assume most
users are migrating from an existing Hadoop deployment, so HDFS is
sort of assumed.


On 20 October 2014 23:18, CCAAT <ccaat@tampabay.rr.com> wrote:
> On 10/20/14 11:46, Steven Schlansker wrote:
>
>
>> We are running Mesos entirely without HDFS with no problems.  We use
>> Docker to distribute our
>> application to slave nodes, and keep no state on individual nodes.
>
>
>
> Background: I'm building up a 3 node cluster to run mesos and spark. No
> legacy Hadoop needed or wanted. I am using btrfs for the local file system,
> with (2) drives set up for raid1 on each system.
>
> So you  are suggesting that I can install mesos + spark + docker
> and not a DFS on these (3) machines?
>
>
> Will I need any other softwares? My application is a geophysical
> fluid simulator, so scala, R, and all sorts of advanced math will
> be required on the cluster for the Finite Element Methods.
>
>
> James
>
>

Mime
View raw message