spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeeyoung Kim (JIRA)" <>
Subject [jira] [Commented] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy
Date Fri, 28 Apr 2017 22:11:04 GMT


Jeeyoung Kim commented on SPARK-10878:

Can't we fix the race condition, at least for the pom-submitting part if we override the function

  /** A nice function to use in tests as well. Values are dummy strings. */
  def getModuleDescriptor: DefaultModuleDescriptor = DefaultModuleDescriptor.newDefaultInstance(
    ModuleRevisionId.newInstance("org.apache.spark", "spark-submit-parent", "1.0"))

to generate random strings?

> Race condition when resolving Maven coordinates via Ivy
> -------------------------------------------------------
>                 Key: SPARK-10878
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.5.0
>            Reporter: Ryan Williams
>            Priority: Minor
> I've recently been shell-scripting the creation of many concurrent Spark-on-YARN apps
and observing a fraction of them to fail with what I'm guessing is a race condition in their
Maven-coordinate resolution.
> For example, I might spawn an app for each path in file {{paths}} with the following
shell script:
> {code}
> cat paths | parallel "$SPARK_HOME/bin/spark-submit foo.jar {}"
> {code}
> When doing this, I observe some fraction of the spawned jobs to fail with errors like:
> {code}
> :: retrieving :: org.apache.spark#spark-submit-parent
>         confs: [default]
> Exception in thread "main" java.lang.RuntimeException: problem during retrieve of org.apache.spark#spark-submit-parent:
java.text.ParseException: failed to parse report: /hpc/users/willir31/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml:
Premature end of file.
>         at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(
>         at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(
>         at org.apache.ivy.Ivy.retrieve(
>         at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1006)
>         at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
>         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.text.ParseException: failed to parse report: /hpc/users/willir31/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml:
Premature end of file.
>         at
>         at org.apache.ivy.core.retrieve.RetrieveEngine.determineArtifactsToCopy(
>         at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(
>         ... 7 more
> Caused by: org.xml.sax.SAXParseException; Premature end of file.
>         at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
>         at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
>         at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
> {code}
> The more apps I try to launch simultaneously, the greater fraction of them seem to fail
with this or similar errors; a batch of ~10 will usually work fine, a batch of 15 will see
a few failures, and a batch of ~60 will have dozens of failures.
> [This gist shows 11 recent failures I observed|].

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message