spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Fregly <ch...@fregly.com>
Subject Re: SPIP: Spark on Kubernetes
Date Fri, 18 Aug 2017 00:59:01 GMT
@reynold:

Databricks runs their proprietary product on Kubernetes.  how about contributing some of that
work back to the Open Source Community?

—

Chris Fregly
Founder and Research Engineer @ PipelineAI <http://pipeline.io/>
Founder @ Advanced Spark and TensorFlow Meetup <http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup>
San Francisco - Chicago - Washington DC - London

> On Aug 17, 2017, at 10:55 AM, Reynold Xin <rxin@databricks.com> wrote:
> 
> +1 on adding Kubernetes support in Spark (as a separate module similar to how YARN is
done)
> 
> I talk with a lot of developers and teams that operate cloud services, and k8s in the
last year has definitely become one of the key projects, if not the one with the strongest
momentum in this space. I'm not 100% sure we can make it into 2.3 but IMO based on the activities
in the forked repo and claims that certain deployments are already running in production,
this could already be a solid project and will have everlasting positive impact.
> 
> 
> 
> On Wed, Aug 16, 2017 at 10:24 AM, Alexander Bezzubov <bzz@apache.org <mailto:bzz@apache.org>>
wrote:
> +1 (non-binding)
> 
> 
> Looking forward using it as part of Apache Spark release, instead of Standalone cluster
deployed on top of k8s.
> 
> 
> --
> Alex
> 
> On Wed, Aug 16, 2017 at 11:11 AM, Ismaël Mejía <iemejia@gmail.com <mailto:iemejia@gmail.com>>
wrote:
> +1 (non-binding)
> 
> This is something really great to have. More schedulers and runtime
> environments are a HUGE win for the Spark ecosystem.
> Amazing work, Big kudos for the guys who created and continue working on this.
> 
> On Wed, Aug 16, 2017 at 2:07 AM, lucas.gary@gmail.com <mailto:lucas.gary@gmail.com>
> <lucas.gary@gmail.com <mailto:lucas.gary@gmail.com>> wrote:
> > From our perspective, we have invested heavily in Kubernetes as our cluster
> > manager of choice.
> >
> > We also make quite heavy use of spark.  We've been experimenting with using
> > these builds (2.1 with pyspark enabled) quite heavily.  Given that we've
> > already 'paid the price' to operate Kubernetes in AWS it seems rational to
> > move our jobs over to spark on k8s.  Having this project merged into the
> > master will significantly ease keeping our Data Munging toolchain primarily
> > on Spark.
> >
> >
> > Gary Lucas
> > Data Ops Team Lead
> > Unbounce
> >
> > On 15 August 2017 at 15:52, Andrew Ash <andrew@andrewash.com <mailto:andrew@andrewash.com>>
wrote:
> >>
> >> +1 (non-binding)
> >>
> >> We're moving large amounts of infrastructure from a combination of open
> >> source and homegrown cluster management systems to unify on Kubernetes and
> >> want to bring Spark workloads along with us.
> >>
> >> On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 <liyinan926@gmail.com <mailto:liyinan926@gmail.com>>
wrote:
> >>>
> >>> +1 (non-binding)
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>> http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22164.html
<http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22164.html>
> >>> Sent from the Apache Spark Developers List mailing list archive at
> >>> Nabble.com.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <mailto:dev-unsubscribe@spark.apache.org>
> >>>
> >>
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <mailto:dev-unsubscribe@spark.apache.org>
> 
> 
> 


Mime
View raw message