spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: spark.task.maxFailures
Date Mon, 16 Dec 2013 23:39:10 GMT
i guess it should really be "maximum number of total task run attempts".
 At least that's what it looks logically. in that sense, the rest of the
documentation is correct ( should be at least 1; 1 = task is allowed no
retries (1-1=0)).




On Fri, Nov 29, 2013 at 2:02 AM, Grega Kešpret <grega@celtra.com> wrote:

> Looking at
> http://spark.incubator.apache.org/docs/latest/configuration.html
> docs says:
> Number of individual task failures before giving up on the job. Should be
> greater than or equal to 1. Number of allowed retries = this value - 1.
>
> However, looking at the code
>
> https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala#L532
>
> if I set spark.task.maxFailures to 1, this means that job will fail after
> task fails for the second time. Shouldn't this line be corrected to if (
> numFailures(index) >= MAX_TASK_FAILURES) {
> ?
>
> I can open a pull request if this is the case.
>
> Thanks,
> Grega
> --
> [image: Inline image 1]
> *Grega Kešpret*
> Analytics engineer
>
> Celtra — Rich Media Mobile Advertising
> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message