spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eran Medan (JIRA)" <>
Subject [jira] [Commented] (SPARK-5023) In Web UI job history, the total job duration is incorrect (much smaller than the sum of its stages)
Date Sun, 26 Apr 2015 04:26:38 GMT


Eran Medan commented on SPARK-5023:

I don't think this is a duplicate, the information shows correctly in "live" view, the incorrect
numbers are for the history / event view. 

I had no lost partitions and no failures, but still - in live view, something that took seconds
or minutes shows as milliseconds or seconds in the history view. I'll debug it and see if
I can figure out the root cause. 

No error messages. 

> In Web UI job history, the total job duration is incorrect (much smaller than the sum
of its stages)
> ----------------------------------------------------------------------------------------------------
>                 Key: SPARK-5023
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.1.1, 1.2.0
>         Environment: Amazon EC2 AMI r3.2xlarge, cluster of 20 to 50 nodes, running the
ec2 provided scripts to create. 
>            Reporter: Eran Medan
> I'm running a long process using Spark + Graph and things look good on the 4040 job status
UI, but when the job is done, when going to the history then the job total duration is much,
much smaller than the total of its stages.
> The way I set logs up is this:
>   val homeDir = sys.props("user.home")
>   val logsPath = new File(homeDir,"sparkEventLogs")
>   val conf = new SparkConf().setAppName("...")
>   conf.set("spark.eventLog.enabled", "true").set("spark.eventLog.dir", logsPath.getCanonicalPath)
> for example job ID X - duration 0.2 s, but when I click the job and look at its stages,
the sum of their duration is more than 15 minutes!
> (before the job was over, in the 4040 job status, the job duration was correct, it is
only incorrect when its done and going to the logs) 
> I hope I didn't configure something because I was very surprised no one reported it yet
(I searched, but perhaps I missed it) 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message