spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-9498) Some statistical information missed when the driver is out of the cluster
Date Sun, 02 Aug 2015 07:59:04 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-9498.
------------------------------
    Resolution: Not A Problem

> Some statistical information missed when the driver is out of the cluster
> -------------------------------------------------------------------------
>
>                 Key: SPARK-9498
>                 URL: https://issues.apache.org/jira/browse/SPARK-9498
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 1.3.1, 1.4.0
>            Reporter: Liang Lee
>
> When an application is submited and the driver is out of the Spark cluster, Some statistical
information missed sometimes. 
> In stage detail inforamtion page, it will display following  info when the driver is
in the spark cluster:
> Details for Stage 7
>     Total task time across all tasks: 37 min
>      Input Size / Records: 55.8 GB / 60488
>      Shuffle write: 26.6 GB / 585242962 
> But when the dreive is out of the spark cluster, it will sometimes  display above info,
while sometimes not, just like this:
> Details for Stage 7
>     Total task time across all tasks: 37 min
> That is the Input Size and Shuffle data does not display.
> I have check the code and find that when the input size is zero then it will not display.

> And the  input size is sent by each Executors and collected by Driver. 
> The problem is that the data that should be repored by Executors, is missed. 
> But I don't know why. Could anyone help to solve this problem?
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message