spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathieu Longtin <math...@closetwork.org>
Subject Re: How to set the degree of parallelism in Spark SQL?
Date Mon, 23 May 2016 12:48:29 GMT
Since the default is 200, I would guess you're only running 2 executors.
Try to verify how many executor you are actually running with the web
interface (port 8080 where the master is running).

On Sat, May 21, 2016 at 11:42 PM Ted Yu <yuzhihong@gmail.com> wrote:

> Looks like an equal sign is missing between partitions and 200.
>
> On Sat, May 21, 2016 at 8:31 PM, SRK <swethakasireddy@gmail.com> wrote:
>
>> Hi,
>>
>> How to set the degree of parallelism in Spark SQL? I am using the
>> following
>> but it somehow seems to allocate only two executors at a time.
>>
>>  sqlContext.sql(" set spark.sql.shuffle.partitions  200  ")
>>
>> Thanks,
>> Swetha
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-set-the-degree-of-parallelism-in-Spark-SQL-tp26996.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
> --
Mathieu Longtin
1-514-803-8977

Mime
View raw message