spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "peay (JIRA)" <>
Subject [jira] [Commented] (SPARK-27019) Spark UI's SQL tab shows inconsistent values
Date Fri, 01 Mar 2019 12:00:00 GMT


peay commented on SPARK-27019:

OK, I can actually reproduce it pretty easily with pyspark:
df_test = spark.range(1024 * 1024 * 1024 * 10).toPandas(){code}
This makes the tasks fail because my executors don't have enough memory, which seems to be
key to hitting the issue. Using only 1000 elements, the job succeeds and it does not trigger
the issue.





> Spark UI's SQL tab shows inconsistent values
> --------------------------------------------
>                 Key: SPARK-27019
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL, Web UI
>    Affects Versions: 2.4.0
>            Reporter: peay
>            Priority: Major
>         Attachments: query-1-details.png, query-1-list.png, query-job-1.png, screenshot-spark-ui-details.png,
> Since 2.4.0, I am frequently seeing broken outputs in the SQL tab of the Spark UI, where
submitted/duration make no sense, description has the ID instead of the actual description.
> Clicking on the link to open a query, the SQL plan is missing as well.
> I have tried to increase `spark.scheduler.listenerbus.eventqueue.capacity` to very large
values like 30k out of paranoia that we may have too many events, but to no avail. I have
not identified anything particular that leads to that: it doesn't occur in all my jobs, but
it does occur in a lot of them still.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message