spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anirudh Ramanathan (JIRA)" <>
Subject [jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates
Date Thu, 31 May 2018 18:18:00 GMT


Anirudh Ramanathan commented on SPARK-24434:

The way several custom APIs have done this before is having a PodTemplate field that uses
the Kubernetes API to provide a rich type-safe interface to add arbitrary modifications to
pods. It's typically easier with golang structs to do that, but we should investigate if from
openapi, there's a way for the Java client to expose the same. Given that we will want it
to map back to stringified configuration, supporting JSON strings seems like a good choice
So, the flow I see is JSON strings converted into valid (type-checked) and supported PodTemplate
specifications that are eventually added to driver and executor pods.

> Support user-specified driver and executor pod templates
> --------------------------------------------------------
>                 Key: SPARK-24434
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes
>    Affects Versions: 2.4.0
>            Reporter: Yinan Li
>            Priority: Major
> With more requests for customizing the driver and executor pods coming, the current approach
of adding new Spark configuration options has some serious drawbacks: 1) it means more Kubernetes
specific configuration options to maintain, and 2) it widens the gap between the declarative
model used by Kubernetes and the configuration model used by Spark. We should start designing
a solution that allows users to specify pod templates as central places for all customization
needs for the driver and executor pods. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message