spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurvinder Singh <gurvinder.si...@uninett.no>
Subject Re: mesos or kubernetes ?
Date Sun, 14 Aug 2016 10:57:39 GMT
On 08/13/2016 08:24 PM, guyoh wrote:
> My company is trying to decide whether to use kubernetes or mesos. Since we
> are planning to use Spark in the near future, I was wandering what is the
> best choice for us. 
> Thanks, 
> Guy
> 
Both Kubernetes and Mesos enables you to share your infrastructure with
other workloads. In K8s it is mainly standalone mode and to provide
failure recovery you can use the options mentioned here
https://spark.apache.org/docs/latest/spark-standalone.html#high-availability
I have tested file system recovery with K8s and it works fine, as K8s
restart the master with in few seconds. In K8s you can dynamically scale
your cluster as K8s support horizontal pod autoscaling
http://blog.kubernetes.io/2016/03/using-Spark-and-Zeppelin-to-process-Big-Data-on-Kubernetes.html
which helps you spin down your spark cluster when not is use and you can
run other workloads using docker/rkt containers on your resources and
scale up spark cluster when needed given you have free resources. You
don't need static partitioning for your spark cluster when running on
k8s, as spark will run in containers and will share the same underlying
resources as others.

You can soonish (https://github.com/apache/spark/pull/13950) use the
authentication based on Github,Google,FB etc to access Spark master UI
when deployed in standalone e.g. on K8s and only expose the master UI to
access all other UIs (worker logs, app).

As usual choice is yours, I just wanted to give you the info about k8s side.

- Gurvinder
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/mesos-or-kubernetes-tp27530.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message