spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners
Date Tue, 22 May 2018 13:04:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan resolved SPARK-20087.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.4.0

Issue resolved by pull request 21165
[https://github.com/apache/spark/pull/21165]

> Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-20087
>                 URL: https://issues.apache.org/jira/browse/SPARK-20087
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.1.0
>            Reporter: Charles Lewis
>            Assignee: Xianjin YE
>            Priority: Major
>             Fix For: 2.4.0
>
>
> When tasks end due to an ExceptionFailure, subscribers to onTaskEnd receive accumulators
/ task metrics for that task, if they were still available. These metrics are not currently
sent when tasks are killed intentionally, such as when a speculative retry finishes, and the
original is killed (or vice versa). Since we're killing these tasks ourselves, these metrics
should almost always exist, and we should treat them the same way as we treat ExceptionFailures.
> Sending these metrics with the TaskKilled end reason makes aggregation across all tasks
in an app more accurate. This data can inform decisions about how to tune the speculation
parameters in order to minimize duplicated work, and in general, the total cost of an app
should include both successful and failed tasks, if that information exists.
> PR: https://github.com/apache/spark/pull/17422



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message