spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Okolnychyi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-23638) Spark on k8s: spark.kubernetes.initContainer.image has no effect
Date Sun, 11 Mar 2018 19:33:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394615#comment-16394615
] 

Anton Okolnychyi commented on SPARK-23638:
------------------------------------------

Could you also share your spark-submit command? It looks you are specifying a custom docker
image for the init container (as ``spark.kubernetes.initContainer.image`` is different from ``spark.kubernetes.container.image``).
Are you sure you need a custom docker image for the init container?

In general, if you have a remote jar in --jars and specify ``spark.kubernetes.container.image``,
Spark will create an init container for you and you do not need to reason about it.

> Spark on k8s: spark.kubernetes.initContainer.image has no effect
> ----------------------------------------------------------------
>
>                 Key: SPARK-23638
>                 URL: https://issues.apache.org/jira/browse/SPARK-23638
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>         Environment: K8 server: Ubuntu 16.04
> Submission client: macOS Sierra 10.12.x
> Client Version: version.Info\{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b",
GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc",
Platform:"darwin/amd64"}
> Server Version: version.Info\{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd",
GitTreeState:"clean", BuildDate:"2017-11-08T18:27:48Z", GoVersion:"go1.8.3", Compiler:"gc",
Platform:"linux/amd64"}
>            Reporter: maheshvra
>            Priority: Major
>
> Hi all - I am trying to use initContainer to download remote dependencies. To begin with,
I ran a test with initContainer which basically "echo hello world". However, when i triggered
the pod deployment via spark-submit, I did not see any trace of initContainer execution in
my kubernetes cluster.
>  
> {code:java}
> SPARK_DRIVER_MEMORY: 1g 
> SPARK_DRIVER_CLASS: com.bigdata.App SPARK_DRIVER_ARGS: -c /opt/spark/work-dir/app/main/environments/int
-w ./../../workflows/workflow_main.json -e prod -n features -v off 
> SPARK_DRIVER_BIND_ADDRESS:  
> SPARK_JAVA_OPT_0: -Dspark.submit.deployMode=cluster 
> SPARK_JAVA_OPT_1: -Dspark.driver.blockManager.port=7079 
> SPARK_JAVA_OPT_2: -Dspark.app.name=fg-am00-raw12 
> SPARK_JAVA_OPT_3: 
> -Dspark.kubernetes.container.image=docker.com/cmapp/fg-am00-raw:1.0.0 
> SPARK_JAVA_OPT_4: -Dspark.app.id=spark-4fa9a5ce1b1d401fa9c1e413ff030d44 SPARK_JAVA_OPT_5: -Dspark.jars=/opt/spark/jars/aws-java-sdk-1.7.4.jar,/opt/spark/jars/hadoop-aws-2.7.3.jar,/opt/spark/jars/guava-14.0.1.jar,/opt/spark/jars/SparkApp.jar,/opt/spark/jars/datacleanup-component-1.0-SNAPSHOT.jar

> SPARK_JAVA_OPT_6: -Dspark.driver.port=7078 
> SPARK_JAVA_OPT_7: -Dspark.kubernetes.initContainer.image=docker.com/cmapp/custombusybox:1.0.0
SPARK_JAVA_OPT_8: -Dspark.kubernetes.executor.podNamePrefix=fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615

> SPARK_JAVA_OPT_9: -Dspark.kubernetes.driver.pod.name=fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615-driver

> SPARK_JAVA_OPT_10: -Dspark.driver.host=fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615-driver-svc.experimental.svc
SPARK_JAVA_OPT_11: -Dspark.executor.instances=5 
> SPARK_JAVA_OPT_12: -Dspark.hadoop.fs.s3a.server-side-encryption-algorithm=AES256 SPARK_JAVA_OPT_13: -Dspark.kubernetes.namespace=experimental
SPARK_JAVA_OPT_14: -Dspark.kubernetes.authenticate.driver.serviceAccountName=experimental-service-account
SPARK_JAVA_OPT_15: -Dspark.master=k8s://https://bigdata
> {code}
>  
> Further, I did not see spec.initContainers section in the generated pod. Please see the
details below
>  
> {code:java}
>  
> {
> "kind": "Pod",
> "apiVersion": "v1",
> "metadata": {
> "name": "fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615-driver",
> "namespace": "experimental",
> "selfLink": "/api/v1/namespaces/experimental/pods/fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615-driver",
> "uid": "adc5a50a-2342-11e8-87dc-12c5b3954044",
> "resourceVersion": "299054",
> "creationTimestamp": "2018-03-09T02:36:32Z",
> "labels": {
> "spark-app-selector": "spark-4fa9a5ce1b1d401fa9c1e413ff030d44",
> "spark-role": "driver"
> },
> "annotations": {
> "spark-app-name": "fg-am00-raw12"
> }
> },
> "spec": {
> "volumes": [
> {
> "name": "experimental-service-account-token-msmth",
> "secret": {
> "secretName": "experimental-service-account-token-msmth",
> "defaultMode": 420
> }
> }
> ],
> "containers": [
> {
> "name": "spark-kubernetes-driver",
> "image": "docker.com/cmapp/fg-am00-raw:1.0.0",
> "args": [
> "driver"
> ],
> "env": [
> {
> "name": "SPARK_DRIVER_MEMORY",
> "value": "1g"
> },
> {
> "name": "SPARK_DRIVER_CLASS",
> "value": "com.myapp.App"
> },
> {
> "name": "SPARK_DRIVER_ARGS",
> "value": "-c /opt/spark/work-dir/app/main/environments/int -w ./../../workflows/workflow_main.json
-e prod -n features -v off"
> },
> {
> "name": "SPARK_DRIVER_BIND_ADDRESS",
> "valueFrom": {
> "fieldRef": {
> "apiVersion": "v1",
> "fieldPath": "status.podIP"
> }
> }
> },
> {
> "name": "SPARK_MOUNTED_CLASSPATH",
> "value": "/opt/spark/jars/aws-java-sdk-1.7.4.jar:/opt/spark/jars/hadoop-aws-2.7.3.jar:/opt/spark/jars/guava-14.0.1.jar:/opt/spark/jars/datacleanup-component-1.0-SNAPSHOT.jar:/opt/spark/jars/SparkApp.jar"
> },
> {
> "name": "SPARK_JAVA_OPT_0",
> "value": "-Dspark.submit.deployMode=cluster"
> },
> {
> "name": "SPARK_JAVA_OPT_1",
> "value": "-Dspark.driver.blockManager.port=7079"
> },
> {
> "name": "SPARK_JAVA_OPT_2",
> "value": "-Dspark.app.name=fg-am00-raw12"
> },
> {
> "name": "SPARK_JAVA_OPT_3",
> "value": "-Dspark.kubernetes.container.image=docker.com/cmapp/fg-am00-raw:1.0.0"
> },
> {
> "name": "SPARK_JAVA_OPT_4",
> "value": "-Dspark.app.id=spark-4fa9a5ce1b1d401fa9c1e413ff030d44"
> },
> {
> "name": "SPARK_JAVA_OPT_5",
> "value": "-Dspark.jars=/opt/spark/jars/aws-java-sdk-1.7.4.jar,/opt/spark/jars/hadoop-aws-2.7.3.jar,/opt/spark/jars/guava-14.0.1.jar,/opt/spark/jars/SparkApp.jar,/opt/spark/jars/datacleanup-component-1.0-SNAPSHOT.jar"
> },
> {
> "name": "SPARK_JAVA_OPT_6",
> "value": "-Dspark.driver.port=7078"
> },
> {
> "name": "SPARK_JAVA_OPT_7",
> "value": "-Dspark.kubernetes.initContainer.image=docker.com/cmapp/custombusybox:1.0.0"
> },
> {
> "name": "SPARK_JAVA_OPT_8",
> "value": "-Dspark.kubernetes.executor.podNamePrefix=fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615"
> },
> {
> "name": "SPARK_JAVA_OPT_9",
> "value": "-Dspark.kubernetes.driver.pod.name=fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615-driver"
> },
> {
> "name": "SPARK_JAVA_OPT_10",
> "value": "-Dspark.driver.host=fg-am00-raw12-b1c8112b8536304ab0fc64fcc41e0615-driver-svc.experimental.svc"
> },
> {
> "name": "SPARK_JAVA_OPT_11",
> "value": "-Dspark.executor.instances=5"
> },
> {
> "name": "SPARK_JAVA_OPT_12",
> "value": "-Dspark.hadoop.fs.s3a.server-side-encryption-algorithm=AES256"
> },
> {
> "name": "SPARK_JAVA_OPT_13",
> "value": "-Dspark.kubernetes.namespace=experimental"
> },
> {
> "name": "SPARK_JAVA_OPT_14",
> "value": "-Dspark.kubernetes.authenticate.driver.serviceAccountName=experimental-service-account"
> },
> {
> "name": "SPARK_JAVA_OPT_15",
> "value": "-Dspark.master=k8s://https://bigdata.com"
> }
> ],
> "resources": {
> "limits": {
> "memory": "1408Mi"
> },
> "requests": {
> "cpu": "1",
> "memory": "1Gi"
> }
> },
> "volumeMounts": [
> {
> "name": "experimental-service-account-token-msmth",
> "readOnly": true,
> "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
> }
> ],
> "terminationMessagePath": "/dev/termination-log",
> "terminationMessagePolicy": "File",
> "imagePullPolicy": "IfNotPresent"
> }
> ],
> "restartPolicy": "Never",
> "terminationGracePeriodSeconds": 30,
> "dnsPolicy": "ClusterFirst",
> "serviceAccountName": "experimental-service-account",
> "serviceAccount": "experimental-service-account",
> "nodeName": "192-12-18-19.bigdata.com",
> "securityContext": {},
> "schedulerName": "default-scheduler"
> },
> "status": {
> "phase": "Succeeded",
> "conditions": [
> {
> "type": "Initialized",
> "status": "True",
> "lastProbeTime": null,
> "lastTransitionTime": "2018-03-09T02:36:32Z",
> "reason": "PodCompleted"
> },
> {
> "type": "Ready",
> "status": "False",
> "lastProbeTime": null,
> "lastTransitionTime": "2018-03-09T02:37:06Z",
> "reason": "PodCompleted"
> },
> {
> "type": "PodScheduled",
> "status": "True",
> "lastProbeTime": null,
> "lastTransitionTime": "2018-03-09T02:36:32Z"
> }
> ],
> "hostIP": "10.205.95.107",
> "podIP": "10.10.158.104",
> "startTime": "2018-03-09T02:36:32Z",
> "containerStatuses": [
> {
> "name": "spark-kubernetes-driver",
> "state": {
> "terminated": {
> "exitCode": 0,
> "reason": "Completed",
> "startedAt": "2018-03-09T02:36:33Z",
> "finishedAt": "2018-03-09T02:37:06Z",
> "containerID": "docker://3833792b943b564c64dc93f15221c30bb05dff9610c46afe8d329cd40ac24505"
> }
> },
> "lastState": {},
> "ready": false,
> "restartCount": 0,
> "image": "docker.com/myapp/fg-am00-raw:1.0.0",
> "imageID": "docker-pullable://docker.com/cmapp/fg-am00-raw@sha256:40ad40c563de50e956059e3072a480176439f07e0c239d07710e69107e3df379",
> "containerID": "docker://3833792b943b564c64dc93f15221c30bb05dff9610c46afe8d329cd40ac24505"
> }
> ],
> "qosClass": "Burstable"
> }
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message