spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <>
Subject Re: SparkSQL Performance Tuning Options
Date Wed, 28 Jan 2015 07:43:24 GMT

On 1/27/15 5:55 PM, Cheng Lian wrote:
> On 1/27/15 11:38 AM, Manoj Samel wrote:
>> Spark 1.2, no Hive, prefer not to use HiveContext to avoid metastore_db.
>> Use case is Spark Yarn app will start and serve as query server for 
>> multiple users i.e. always up and running. At startup, there is 
>> option to cache data and also pre-compute some results sets, hash 
>> maps etc. that would be likely be asked by client APIs. I.e there is 
>> some option to use startup time to precompute/cache - but query 
>> response time requirement on large data set is very stringent
>> Hoping to use SparkSQL (but a combination of SQL and RDD APIs is also 
>> OK).
>> * Does SparkSQL execution uses underlying partition information ? 
>> (Data is from HDFS)
> No. For example, if the underlying data has already been partitioned 
> by some key, Spark SQL doesn't know it, and can't leverage that 
> information to avoid shuffle when doing aggregation on that key. 
> However, partitioning the data ahead of time does help minimizing 
> shuffle network IO. There's a JIRA ticket to enable Spark SQL aware of 
> underlying data distribution.

Maybe you are asking about locality? If that's the case, just want to 
add that Spark SQL does understand locality information of the 
underlying data. It's obtained from Hadoop InputFormat.

>> * Are there any ways to give "hints" to the SparkSQL execution about 
>> any precomputed/pre-cached RDDs?
> Instead of caching raw RDD, it's recommended to transform raw RDD to 
> SchemaRDD and then cache it, so that in-memory columnar storage can be 
> used. Also Spark SQL recognizes cached SchemaRDDs automatically.
>> * Packages spark.sql.execution, spark.sql.execution.joins and other 
>> packages - would using these for tuning query plan is 
>> recommended? Would like to keep this as-needed if possible
> Not sure whether I understood this question. Are you trying to use 
> internal APIs to do customized optimizations?
>> * Features not in current release but scheduled for upcoming release 
>> would also be good to know.
>> Thanks,
>> PS: This is not a small topic so if someone prefers to start a 
>> offline thread on details, I can do that and summarize the 
>> conclusions back to this thread.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message