mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikram Kone <>
Subject Can mesos provide high availability for any generic app/framework on cassandra cluster?
Date Mon, 17 Aug 2015 21:46:54 GMT
I'm looking at existing open source workflow engines we can use for
scheduling spark jobs with intricate dependencies on a datastax cassandra
cluster. Currently we  are using crontab to schedule jobs and want to move
to something which is more robust and highly available.
There are 2 main problems with cron on cassandra
​as we have today
1. Single point of failure: Our cron tasks that do spark-submit run on a
single machine and if that machine goes down in the cluster, all the jobs
are kaput till the node comes back up.
2. Can't easily specify job dependency between cron tasks to model a DAG.

One of the
​Work flow ​
engines I'm looking at is
zkaban where
​ ​
job authoring and dependency config is easy
​ via Web UI and REST APIs​
. But it
has a single point of failure for azkaban master . I'm also open to run the
workflow engine on a separate cluster but since spark doesn't allow remote
job submission
​s natively​
, we are stuck with running workflow engine on the same cassandra cluster.

​High availability of Spark master is taken care of in the Datastax's
version of cassandra. So all I need to do is provide HA for Azkaban. Is
mesos the right tool for this?

I can either go Spark + Mesos + Zookeper  if...

 Mesos provides the ability to configure jobs with dependencies ie run job
A after Job and job C are finished.

Or go with
Spark + Azkaban + zookeper if..
Mesos doesn't provide job dependency features.



View raw message