spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <luohui20...@sina.com>
Subject 回复:Re: sparksql running slow while joining 2 tables.
Date Mon, 04 May 2015 13:07:56 GMT
hi Olivierspark1.3.1, with java1.8.0.45and add 2 pics .it seems like a GC issue. I also tried
with different parameters like memory size of driver&executor, memory fraction, java opts...but
this issue still happens.

--------------------------------
 
Thanks&amp;Best regards!
罗辉 San.Luo

----- 原始邮件 -----
发件人:Olivier Girardot <ssaboum@gmail.com>
收件人:luohui20001@sina.com, user <user@spark.apache.org>
主题:Re: sparksql running slow while joining 2 tables.
日期:2015年05月04日 20点46分

Hi, 
What is you Spark version ?

Regards, 
Olivier.
Le lun. 4 mai 2015 à 11:03, <luohui20001@sina.com> a écrit :
hi guys        when i am running a sql  like "select a.name,a.startpoint,a.endpoint, a.piece
from db a join sample b on (a.name = b.name) where (b.startpoint > a.startpoint + 25);"
I found sparksql running slow in minutes which may caused by very long GC and shuffle time.
       table db is created from a txt file size at 56mb while table sample sized at 26mb,
both at small size.       my spark cluster is a standalone  pseudo-distributed spark cluster
with 8g executor and 4g driver manager.       any advises? thank you guys. 
--------------------------------
 
Thanks&amp;Best regards!
罗辉 San.Luo


---------------------------------------------------------------------

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

For additional commands, e-mail: user-help@spark.apache.org

Mime
View raw message