airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Laird <br.la...@gmail.com>
Subject Re: --archives flag missing from SparkSubmitHook
Date Mon, 17 Sep 2018 17:15:23 GMT
Looking at this again, I think one could just use set
`spark.yarn.dist.archives` in the conf passed to the job. If that works,
please disregard :)

On Mon, Sep 17, 2018 at 1:09 PM Ben Laird <br.laird@gmail.com> wrote:

> The current SparkSubmitHook doesn't appear to support the --archives flag.
> From the Spark docs:
>
> "spark.yarn.dist.archives (none): Comma separated list of archives to be
> extracted into the working directory of each executor."
> https://spark.apache.org/docs/latest/running-on-yarn.html
>
> This is necessary for deploying zipped virtualenvs or other packages
> across the cluster. For now, I'll have to maintain my own copy of this Hook
> but will contribute this back if others aren't planning on it.
>
> Curious if there is context on why this isn't included?
>
> Ben
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message