spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakub Dubovsky (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23283) Probable mistake in hasLaunchedTask condition
Date Wed, 31 Jan 2018 13:47:00 GMT
Jakub Dubovsky created SPARK-23283:
--------------------------------------

             Summary: Probable mistake in hasLaunchedTask condition
                 Key: SPARK-23283
                 URL: https://issues.apache.org/jira/browse/SPARK-23283
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.2.1
            Reporter: Jakub Dubovsky


In class {code}TaskSchedulerImpl{code} there is a [variable|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L97]
tracking whether some task was already launched. It is then set to true [here|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L377]
whenever some task is launched.

I believe the [condition|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L376]
should be {code}tasks.map(_.size).sum > 0{code} instead of current {code}tasks.size >
0{code}

Current condition asks whether there are some resource offers while it should ask whether
there were some launched tasks. In some cases it might happen that resources are offered but
no tasks launched. In such a case no warning is issued.

Can you confirm this is valid?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message