spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang-Chi Hsieh (JIRA)" <>
Subject [jira] [Commented] (SPARK-28761) spark.driver.maxResultSize only applies to compressed data
Date Fri, 16 Aug 2019 21:37:00 GMT


Liang-Chi Hsieh commented on SPARK-28761:

If you do it at SparkPlan.scala#L344, isn't it just for SQL? {{spark.driver.maxResultSize}}
covers RDD, right?

> spark.driver.maxResultSize only applies to compressed data
> ----------------------------------------------------------
>                 Key: SPARK-28761
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: David Vogelbacher
>            Priority: Major
> Spark has a setting {{spark.driver.maxResultSize}}, see
> {noformat}
> Limit of total size of serialized results of all partitions for each Spark action (e.g.
collect) in bytes. Should be at least 1M, or 0 for unlimited. 
> Jobs will be aborted if the total size is above this limit. Having a high limit may cause
out-of-memory errors in driver (depends on spark.driver.memory and memory overhead of objects
in JVM). 
> Setting a proper limit can protect the driver from out-of-memory errors.
> {noformat}
> This setting can be very useful in constraining the memory that the spark driver needs
for a specific spark action. However, this limit is checked before decompressing data in
> Even if the compressed data is below the limit the uncompressed data can still be far
above. In order to protect the driver we should also impose a limit on the uncompressed data.
We could do this in
> I propose adding a new config option {{spark.driver.maxUncompressedResultSize}}.
> A simple repro of this with spark shell:
> {noformat}
> > printf 'a%.0s' {1..100000} > test.csv # create a 100 MB file
> > ./bin/spark-shell --conf "spark.driver.maxResultSize=10000"
> scala> val df ="csv").load("/Users/dvogelbacher/test.csv")
> df: org.apache.spark.sql.DataFrame = [_c0: string]
> scala> val results = df.collect()
> results: Array[org.apache.spark.sql.Row] = Array([aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
> scala> results(0).getString(0).size
> res0: Int = 100000
> {noformat}
> Even though we set maxResultSize to 10 MB, we collect a result that is 100MB uncompressed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message