spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] dongjoon-hyun commented on a change in pull request #29897: [SPARK-33006][K8S][DOCS] Add dynamic PVC usage example into K8s doc
Date Wed, 30 Sep 2020 06:51:45 GMT

dongjoon-hyun commented on a change in pull request #29897:
URL: https://github.com/apache/spark/pull/29897#discussion_r497279320



##########
File path: docs/running-on-kubernetes.md
##########
@@ -307,7 +307,18 @@ And, the claim name of a `persistentVolumeClaim` with volume name `checkpointpvc
 spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=check-point-pvc-claim
 ```
 
-The configuration properties for mounting volumes into the executor pods use prefix `spark.kubernetes.executor.`
instead of `spark.kubernetes.driver.`. For a complete list of available options for each supported
type of volumes, please refer to the [Spark Properties](#spark-properties) section below.
+The configuration properties for mounting volumes into the executor pods use prefix `spark.kubernetes.executor.`
instead of `spark.kubernetes.driver.`.
+
+For example, you can mount a dynamically-created persistent volume claim per executor by
using `OnDemand` as a claim name and `storageClass` and `sizeLimit` options like the following.
This is useful in case of [Dynamic Allocation](configuration.html#dynamic-allocation).

Review comment:
       @dbtsai . What do you mean by the following?
   >  Currently, this doesn't support DA yet. 
   
   Since Apache Spark 3.0.0, dynamic allocation with K8s has been supported with shuffle data
tracking. And, this feature is also developed for both additional large disk requirement and
dynamic allocation scenario. For example, in case of dynamic allocation, the executor id increases
monotonically and indefinitely, so users cannot prepare pre-populated PVCs.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message