spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Updated] (SPARK-8475) SparkSubmit with Ivy jars is very slow to load with no internet access
Date Sat, 04 Jul 2015 07:28:05 GMT


Sean Owen updated SPARK-8475:
    Fix Version/s:     (was: 1.4.2)

> SparkSubmit with Ivy jars is very slow to load with no internet access
> ----------------------------------------------------------------------
>                 Key: SPARK-8475
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Submit
>    Affects Versions: 1.4.0
>            Reporter: Nathan McCarthy
>            Assignee: Burak Yavuz
>            Priority: Minor
>             Fix For: 1.4.1, 1.5.0
> Spark Submit adds maven central & spark bintray to the ChainResolver before it adds
any external resolvers. 
> When running on a cluster without internet access, this means the spark shell takes forever
to launch as it tries these two remote repos before the ones specified in the --repositories
list. In our case we have a proxy which the cluster can access it and supply it via --repositories.
> This is also a problem for users who maintain a proxy for maven/ivy repos with something
like Nexus/Artifactory. Having a repo proxy is popular at many organisations so I'd say this
would be a useful change for these users as well. In the current state even if a maven central
proxy is supplied, it will still try and hit central. 
> I see two options for a fix;
> * Change the order repos are added to the ChainResolver, making the --repositories supplied
repos come before anything else.

> * Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default true) which
when false wont add the maven central & bintry to the ChainResolver. 
> Happy to do a PR for this fix. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message