spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Resolved] (SPARK-14836) Zip local jars before uploading to distributed cache
Date Thu, 28 Apr 2016 23:40:12 GMT


Marcelo Vanzin resolved SPARK-14836.
       Resolution: Fixed
         Assignee: Saisai Shao
    Fix Version/s: 2.0.0

> Zip local jars before uploading to distributed cache
> ----------------------------------------------------
>                 Key: SPARK-14836
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 2.0.0
>            Reporter: Saisai Shao
>            Assignee: Saisai Shao
>            Priority: Minor
>             Fix For: 2.0.0
> Currently if neither {{spark.yarn.jars}} nor {{spark.yarn.archive}} is set (by default),
Spark on yarn code will upload all the jars in the folder separately into distributed cache,
this is quite time consuming, and very verbose, instead of upload jars separately into distributed
cache, here changes to zip all the jars first, and then put into distributed cache.
> This will significantly improve the speed of starting time, in my local machine, it could
save around 5 seconds for the starting period, not to say a real cluster. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message