spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Chammas (JIRA)" <>
Subject [jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)
Date Fri, 20 May 2016 02:45:13 GMT


Nicholas Chammas commented on SPARK-3821:

You can deploy Spark today on Docker just fine. It's just that Spark itself does not maintain
any official Dockerfiles and likely never will since the project is actually trying to push
deployment stuff outside the main project (hence why spark-ec2 was moved out; you will not
see spark-ec2 in the official docs once Spark 2.0 comes out). You may be more interested in
the Apache Big Top project, which focuses on big data system deployment (including Spark)
and may have stuff for Docker specifically. 

Mesos is a separate matter, because it's a resource manager (analogous to YARN) that integrates
with Spark at a low level.

If you still think Spark should host and maintain an official Dockerfile and Docker images
that are suitable for production use, please open a separate issue. I think the maintainers
will reject it on the grounds that I have explained here, though. (Can't say for sure; after
all I'm just a random contributor.)

> Develop an automated way of creating Spark images (AMI, Docker, and others)
> ---------------------------------------------------------------------------
>                 Key: SPARK-3821
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build, EC2
>            Reporter: Nicholas Chammas
>            Assignee: Nicholas Chammas
>         Attachments: packer-proposal.html
> Right now the creation of Spark AMIs or Docker containers is done manually. With tools
like [Packer|], we should be able to automate this work, and do so in
such a way that multiple types of machine images can be created from a single template.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message