spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: SPIP: Spark on Kubernetes
Date Tue, 15 Aug 2017 17:09:32 GMT
+1 (non-binding)

I (personally) think that Kubernetes as a scheduler backend should
eventually get merged in and there is clearly a community interested in the
work required to maintain it.

On Tue, Aug 15, 2017 at 9:51 AM William Benton <willb@redhat.com> wrote:

> +1 (non-binding)
>
> On Tue, Aug 15, 2017 at 10:32 AM, Anirudh Ramanathan <
> foxish@google.com.invalid> wrote:
>
>> Spark on Kubernetes effort has been developed separately in a fork, and
>> linked back from the Apache Spark project as an experimental backend
>> <http://spark.apache.org/docs/latest/cluster-overview.html#cluster-manager-types>.
>> We're ~6 months in, have had 5 releases
>> <https://github.com/apache-spark-on-k8s/spark/releases>.
>>
>>    - 2 Spark versions maintained (2.1, and 2.2)
>>    - Extensive integration testing and refactoring efforts to maintain
>>    code quality
>>    - Developer
>>    <https://github.com/apache-spark-on-k8s/spark#getting-started> and
>>    user-facing <https://apache-spark-on-k8s.github.io/userdocs/>
>>     documentation
>>    - 10+ consistent code contributors from different organizations
>>    <https://apache-spark-on-k8s.github.io/userdocs/contribute.html#project-contributions>
involved
>>    in actively maintaining and using the project, with several more members
>>    involved in testing and providing feedback.
>>    - The community has delivered several talks on Spark-on-Kubernetes
>>    generating lots of feedback from users.
>>    - In addition to these, we've seen efforts spawn off such as:
>>    - HDFS on Kubernetes
>>       <https://github.com/apache-spark-on-k8s/kubernetes-HDFS> with
>>       Locality and Performance Experiments
>>       - Kerberized access
>>       <https://docs.google.com/document/d/1RBnXD9jMDjGonOdKJ2bA1lN4AAV_1RwpU_ewFuCNWKg/edit>
to
>>       HDFS from Spark running on Kubernetes
>>
>> *Following the SPIP process, I'm putting this SPIP up for a vote.*
>>
>>    - +1: Yeah, let's go forward and implement the SPIP.
>>    - +0: Don't really care.
>>    - -1: I don't think this is a good idea because of the following
>>    technical reasons.
>>
>> If there is any further clarification desired, on the design or the
>> implementation, please feel free to ask questions or provide feedback.
>>
>>
>> SPIP: Kubernetes as A Native Cluster Manager
>>
>> Full Design Doc: link
>> <https://issues.apache.org/jira/secure/attachment/12881586/SPARK-18278%20Spark%20on%20Kubernetes%20Design%20Proposal%20Revision%202%20%281%29.pdf>
>>
>> JIRA: https://issues.apache.org/jira/browse/SPARK-18278
>>
>> Kubernetes Issue: https://github.com/kubernetes/kubernetes/issues/34377
>>
>> Authors: Yinan Li, Anirudh Ramanathan, Erik Erlandson, Andrew Ash, Matt
>> Cheah,
>>
>> Ilan Filonenko, Sean Suchter, Kimoon Kim
>> Background and Motivation
>>
>> Containerization and cluster management technologies are constantly
>> evolving in the cluster computing world. Apache Spark currently implements
>> support for Apache Hadoop YARN and Apache Mesos, in addition to providing
>> its own standalone cluster manager. In 2014, Google announced development
>> of Kubernetes <https://kubernetes.io/> which has its own unique feature
>> set and differentiates itself from YARN and Mesos. Since its debut, it has
>> seen contributions from over 1300 contributors with over 50000 commits.
>> Kubernetes has cemented itself as a core player in the cluster computing
>> world, and cloud-computing providers such as Google Container Engine,
>> Google Compute Engine, Amazon Web Services, and Microsoft Azure support
>> running Kubernetes clusters.
>>
>> This document outlines a proposal for integrating Apache Spark with
>> Kubernetes in a first class way, adding Kubernetes to the list of cluster
>> managers that Spark can be used with. Doing so would allow users to share
>> their computing resources and containerization framework between their
>> existing applications on Kubernetes and their computational Spark
>> applications. Although there is existing support for running a Spark
>> standalone cluster on Kubernetes
>> <https://github.com/kubernetes/examples/blob/master/staging/spark/README.md>,
>> there are still major advantages and significant interest in having native
>> execution support. For example, this integration provides better support
>> for multi-tenancy and dynamic resource allocation. It also allows users to
>> run applications of different Spark versions of their choices in the same
>> cluster.
>>
>> The feature is being developed in a separate fork
>> <https://github.com/apache-spark-on-k8s/spark> in order to minimize risk
>> to the main project during development. Since the start of the development
>> in November of 2016, it has received over 100 commits from over 20
>> contributors and supports two releases based on Spark 2.1 and 2.2
>> respectively. Documentation is also being actively worked on both in the
>> main project repository and also in the repository
>> https://github.com/apache-spark-on-k8s/userdocs. Regarding real-world
>> use cases, we have seen cluster setup that uses 1000+ cores. We are also
>> seeing growing interests on this project from more and more organizations.
>>
>> While it is easy to bootstrap the project in a forked repository, it is
>> hard to maintain it in the long run because of the tricky process of
>> rebasing onto the upstream and lack of awareness in the large Spark
>> community. It would be beneficial to both the Spark and Kubernetes
>> community seeing this feature being merged upstream. On one hand, it gives
>> Spark users the option of running their Spark workloads along with other
>> workloads that may already be running on Kubernetes, enabling better
>> resource sharing and isolation, and better cluster administration. On the
>> other hand, it gives Kubernetes a leap forward in the area of large-scale
>> data processing by being an officially supported cluster manager for Spark.
>> The risk of merging into upstream is low because most of the changes are
>> purely incremental, i.e., new Kubernetes-aware implementations of existing
>> interfaces/classes in Spark core are introduced. The development is also
>> concentrated in a single place at resource-managers/kubernetes
>> <https://github.com/apache-spark-on-k8s/spark/tree/branch-2.2-kubernetes/resource-managers/kubernetes>.
>> The risk is further reduced by a comprehensive integration test framework,
>> and an active and responsive community of future maintainers.
>> Target Personas
>>
>> Devops, data scientists, data engineers, application developers, anyone
>> who can benefit from having Kubernetes
>> <https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/> as a
>> native cluster manager for Spark.
>> Goals
>>
>>    -
>>
>>    Make Kubernetes a first-class cluster manager for Spark, alongside
>>    Spark Standalone, Yarn, and Mesos.
>>    -
>>
>>    Support both client and cluster deployment mode.
>>    -
>>
>>    Support dynamic resource allocation
>>    <http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation>
>>    .
>>    -
>>
>>    Support Spark Java/Scala, PySpark, and Spark R applications.
>>    -
>>
>>    Support secure HDFS access.
>>    -
>>
>>    Allow running applications of different Spark versions in the same
>>    cluster through the ability to specify the driver and executor Docker
>>    images on a per-application basis.
>>    -
>>
>>    Support specification and enforcement of limits on both CPU cores and
>>    memory.
>>
>> Non-Goals
>>
>>    -
>>
>>    Support cluster resource scheduling and sharing beyond capabilities
>>    offered natively by the Kubernetes per-namespace resource quota model.
>>
>> Proposed API Changes
>>
>> Most API changes are purely incremental, i.e., new Kubernetes-aware
>> implementations of existing interfaces/classes in Spark core are
>> introduced. Detailed changes are as follows.
>>
>>    -
>>
>>    A new cluster manager option KUBERNETES is introduced and some
>>    changes are made to SparkSubmit to make it be aware of this option.
>>    -
>>
>>    A new implementation of CoarseGrainedSchedulerBackend, namely
>>    KubernetesClusterSchedulerBackend is responsible for managing the
>>    creation and deletion of executor Pods through the Kubernetes API.
>>    -
>>
>>    A new implementation of TaskSchedulerImpl, namely
>>    KubernetesTaskSchedulerImpl, and a new implementation of
>>    TaskSetManager, namely Kubernetes TaskSetManager, are introduced for
>>    Kubernetes-aware task scheduling.
>>    -
>>
>>    When dynamic resource allocation is enabled, a new implementation of
>>    ExternalShuffleService, namely KubernetesExternalShuffleService is
>>    introduced.
>>
>> Design Sketch
>>
>> Below we briefly describe the design. For more details on the design and
>> architecture, please refer to the architecture documentation
>> <https://github.com/apache-spark-on-k8s/spark/tree/branch-2.2-kubernetes/resource-managers/kubernetes/architecture-docs>.
>> The main idea of this design is to run Spark driver and executors inside
>> Kubernetes Pods <https://kubernetes.io/docs/concepts/workloads/pods/pod/>.
>> Pods are a co-located and co-scheduled group of one or more containers run
>> in a shared context. The driver is responsible for creating and destroying
>> executor Pods through the Kubernetes API, while Kubernetes is fully
>> responsible for scheduling the Pods to run on available nodes in the
>> cluster. In the cluster mode, the driver also runs in a Pod in the cluster,
>> created through the Kubernetes API by a Kubernetes-aware submission client
>> called by the spark-submit script. Because the driver runs in a Pod, it
>> is reachable by the executors in the cluster using its Pod IP. In the
>> client mode, the driver runs outside the cluster and calls the Kubernetes
>> API to create and destroy executor Pods. The driver must be routable from
>> within the cluster for the executors to communicate with it.
>>
>> The main component running in the driver is the
>> KubernetesClusterSchedulerBackend, an implementation of
>> CoarseGrainedSchedulerBackend, which manages allocating and destroying
>> executors via the Kubernetes API, as instructed by Spark core via calls to
>> methods doRequestTotalExecutors and doKillExecutors, respectively.
>> Within the KubernetesClusterSchedulerBackend, a separate
>> kubernetes-pod-allocator thread handles the creation of new executor
>> Pods with appropriate throttling and monitoring. Throttling is achieved
>> using a feedback loop that makes decision on submitting new requests for
>> executors based on whether previous executor Pod creation requests have
>> completed. This indirection is necessary because the Kubernetes API server
>> accepts requests for new Pods optimistically, with the anticipation of
>> being able to eventually schedule them to run. However, it is undesirable
>> to have a very large number of Pods that cannot be scheduled and stay
>> pending within the cluster. The throttling mechanism gives us control over
>> how fast an application scales up (which can be configured), and helps
>> prevent Spark applications from DOS-ing the Kubernetes API server with too
>> many Pod creation requests. The executor Pods simply run the
>> CoarseGrainedExecutorBackend class from a pre-built Docker image that
>> contains a Spark distribution.
>>
>> There are auxiliary and optional components: ResourceStagingServer and
>> KubernetesExternalShuffleService, which serve specific purposes
>> described below. The ResourceStagingServer serves as a file store (in
>> the absence of a persistent storage layer in Kubernetes) for application
>> dependencies uploaded from the submission client machine, which then get
>> downloaded from the server by the init-containers in the driver and
>> executor Pods. It is a Jetty server with JAX-RS and has two endpoints for
>> uploading and downloading files, respectively. Security tokens are returned
>> in the responses for file uploading and must be carried in the requests for
>> downloading the files. The ResourceStagingServer is deployed as a
>> Kubernetes Service
>> <https://kubernetes.io/docs/concepts/services-networking/service/>
>> backed by a Deployment
>> <https://kubernetes.io/docs/concepts/workloads/controllers/deployment/>
>> in the cluster and multiple instances may be deployed in the same cluster.
>> Spark applications specify which ResourceStagingServer instance to use
>> through a configuration property.
>>
>> The KubernetesExternalShuffleService is used to support dynamic resource
>> allocation, with which the number of executors of a Spark application can
>> change at runtime based on the resource needs. It provides an additional
>> endpoint for drivers that allows the shuffle service to delete driver
>> termination and clean up the shuffle files associated with corresponding
>> application. There are two ways of deploying the
>> KubernetesExternalShuffleService: running a shuffle service Pod on each
>> node in the cluster or a subset of the nodes using a DaemonSet
>> <https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/>,
>> or running a shuffle service container in each of the executor Pods. In the
>> first option, each shuffle service container mounts a hostPath
>> <https://kubernetes.io/docs/concepts/storage/volumes/#hostpath> volume.
>> The same hostPath volume is also mounted by each of the executor
>> containers, which must also have the environment variable
>> SPARK_LOCAL_DIRS point to the hostPath. In the second option, a shuffle
>> service container is co-located with an executor container in each of the
>> executor Pods. The two containers share an emptyDir
>> <https://kubernetes.io/docs/concepts/storage/volumes/#emptydir> volume
>> where the shuffle data gets written to. There may be multiple instances of
>> the shuffle service deployed in a cluster that may be used for different
>> versions of Spark, or for different priority levels with different resource
>> quotas.
>>
>> New Kubernetes-specific configuration options are also introduced to
>> facilitate specification and customization of driver and executor Pods and
>> related Kubernetes resources. For example, driver and executor Pods can be
>> created in a particular Kubernetes namespace and on a particular set of the
>> nodes in the cluster. Users are allowed to apply labels and annotations to
>> the driver and executor Pods.
>>
>> Additionally, secure HDFS support is being actively worked on following
>> the design here
>> <https://docs.google.com/document/d/1RBnXD9jMDjGonOdKJ2bA1lN4AAV_1RwpU_ewFuCNWKg/edit>.
>> Both short-running jobs and long-running jobs that need periodic delegation
>> token refresh are supported, leveraging built-in Kubernetes constructs like
>> Secrets. Please refer to the design doc for details.
>> Rejected DesignsResource Staging by the Driver
>>
>> A first implementation effectively included the ResourceStagingServer in
>> the driver container itself. The driver container ran a custom command that
>> opened an HTTP endpoint and waited for the submission client to send
>> resources to it. The server would then run the driver code after it had
>> received the resources from the submission client machine. The problem with
>> this approach is that the submission client needs to deploy the driver in
>> such a way that the driver itself would be reachable from outside of the
>> cluster, but it is difficult for an automated framework which is not aware
>> of the cluster's configuration to expose an arbitrary pod in a generic way.
>> The Service-based design chosen allows a cluster administrator to expose
>> the ResourceStagingServer in a manner that makes sense for their
>> cluster, such as with an Ingress or with a NodePort service.
>> Kubernetes External Shuffle Service
>>
>> Several alternatives were considered for the design of the shuffle
>> service. The first design postulated the use of long-lived executor pods
>> and sidecar containers in them running the shuffle service. The advantage
>> of this model was that it would let us use emptyDir for sharing as opposed
>> to using node local storage, which guarantees better lifecycle management
>> of storage by Kubernetes. The apparent disadvantage was that it would be a
>> departure from the traditional Spark methodology of keeping executors for
>> only as long as required in dynamic allocation mode. It would additionally
>> use up more resources than strictly necessary during the course of
>> long-running jobs, partially losing the advantage of dynamic scaling.
>>
>> Another alternative considered was to use a separate shuffle service
>> manager as a nameserver. This design has a few drawbacks. First, this means
>> another component that needs authentication/authorization management and
>> maintenance. Second, this separate component needs to be kept in sync with
>> the Kubernetes cluster. Last but not least, most of functionality of this
>> separate component can be performed by a combination of the in-cluster
>> shuffle service and the Kubernetes API server.
>> Pluggable Scheduler Backends
>>
>> Fully pluggable scheduler backends were considered as a more generalized
>> solution, and remain interesting as a possible avenue for future-proofing
>> against new scheduling targets.  For the purposes of this project, adding a
>> new specialized scheduler backend for Kubernetes was chosen as the approach
>> due to its very low impact on the core Spark code; making scheduler fully
>> pluggable would be a high-impact high-risk modification to Spark’s core
>> libraries. The pluggable scheduler backends effort is being tracked in
>> JIRA-19700 <https://issues.apache.org/jira/browse/SPARK-19700>.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
> --
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

Mime
View raw message