spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "igor mazor (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-21252) The duration times showed by spark web UI are inaccurate
Date Thu, 29 Jun 2017 10:43:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068147#comment-16068147
] 

igor mazor edited comment on SPARK-21252 at 6/29/17 10:42 AM:
--------------------------------------------------------------

Yes, its indeed possible to drill to individual stages, however the scenario I described on
my last comment is in the same stage.
The first attempt following by timeout after 20 seconds and then the second successful attempt
are all in the same stage and therefore the UI shows 20 seconds duration, although thats not
true.
Also, if the stage was indeed 40 ms, the UI shows 40 ms as the duration, but with seconds
there is rounding which also causing for  inconsistent result presentation. 
If you already do the rounding, why then the 40 ms is not rounded to 0 seconds ?

Also I noticed now that for example stage that took 42 ms (Job Duration), when I click on
that stage I see Duration = 38 ms and when going deeper into the Summary Metrics for Tasks,
I see that the longest task was 35 ms, so not sure where to all the gaps went.


was (Author: mazor.igal):
Yes, its indeed possible to drill to individual stages, however the scenario I described on
my last comment is happens in the same stage.
The first attempt following by timeout after 20 seconds and then the second successful attempt
are all in the same stage and therefore the UI shows 20 seconds duration, although thats not
true.
Also, if the stage was indeed 40 ms, the UI shows 40 ms as the duration, but with seconds
there is rounding which also causing for  inconsistent result presentation. 
If you already do the rounding, why then the 40 ms is not rounded to 0 seconds ?

Also I noticed now that for example stage that took 42 ms (Job Duration), when I click on
that stage I see Duration = 38 ms and when going deeper into the Summary Metrics for Tasks,
I see that the longest task was 35 ms, so not sure where to all the gaps went.

> The duration times showed by spark web UI are inaccurate
> --------------------------------------------------------
>
>                 Key: SPARK-21252
>                 URL: https://issues.apache.org/jira/browse/SPARK-21252
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1
>            Reporter: igor mazor
>
> The duration times showed by spark UI are inaccurate and seems to be rounded.
> For example when a job had 2 stages, first stage executed in 47 ms and second stage in
3 seconds, the total execution time showed by the UI is 4 seconds.
> Another example, first stage was executed in 20 ms and second stage in 4 seconds, the
total execution time showed by the UI would be in that case also 4 seconds.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message