spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: [SPARK-24547][K8S] Allow for building spark on k8s docker images without cache and don't forget to push spark-py container.
Date Thu, 21 Jun 2018 00:09:53 GMT
Repository: spark
Updated Branches:
  refs/heads/master 3f4bda728 -> 15747cfd3

[SPARK-24547][K8S] Allow for building spark on k8s docker images without cache and don't forget
to push spark-py container.

## What changes were proposed in this pull request?

TL;DR from JIRA issue:

- First time I generated images for 2.4.0 Docker was using it's cache, so actually when running
jobs, old jars where still in the Docker image. This produces errors like this in the executors:

`; local class incompatible:
stream classdesc serialVersionUID = 6155820641931972169, local class serialVersionUID = -3720498261147521051`

- The second problem was that the spark container is pushed, but the spark-py container wasn't
yet. This was just forgotten in the initial PR.

- A third problem I also ran into because I had an older docker was
so I have not included a fix for that in this ticket.

## How was this patch tested?

I've tested it on my own Spark on k8s deployment.

Author: Ray Burgemeestre <>

Closes #21555 from rayburgemeestre/SPARK-24547.


Branch: refs/heads/master
Commit: 15747cfd3246385ffb23e19e28d2e4effa710bf6
Parents: 3f4bda7
Author: Ray Burgemeestre <>
Authored: Wed Jun 20 17:09:37 2018 -0700
Committer: Anirudh Ramanathan <>
Committed: Wed Jun 20 17:09:37 2018 -0700

 bin/ | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/bin/ b/bin/
index a871ab5..a3f1bcf 100755
--- a/bin/
+++ b/bin/
@@ -70,17 +70,18 @@ function build {
   local PYDOCKERFILE=${PYDOCKERFILE:-"$IMG_PATH/spark/bindings/python/Dockerfile"}
-  docker build "${BUILD_ARGS[@]}" \
+  docker build $NOCACHEARG "${BUILD_ARGS[@]}" \
     -t $(image_ref spark) \
-    docker build "${BINDING_BUILD_ARGS[@]}" \
+  docker build $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" \
     -t $(image_ref spark-py) \
     -f "$PYDOCKERFILE" .
 function push {
   docker push "$(image_ref spark)"
+  docker push "$(image_ref spark-py)"
 function usage {
@@ -99,6 +100,7 @@ Options:
   -r repo     Repository address.
   -t tag      Tag to apply to the built image, or to identify the image to be pushed.
   -m          Use minikube's Docker daemon.
+  -n          Build docker image with --no-cache
 Using minikube when building images will do so directly into minikube's Docker daemon.
 There is no need to push the images into minikube in that case, they'll be automatically
@@ -127,7 +129,8 @@ REPO=
-while getopts f:mr:t: option
+while getopts f:mr:t:n option
  case "${option}"
@@ -135,6 +138,7 @@ do
  r) REPO=${OPTARG};;
  t) TAG=${OPTARG};;
+ n) NOCACHEARG="--no-cache";;
    if ! which minikube 1>/dev/null; then
      error "Cannot find minikube."

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message