spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject 回复:Re: sparksql running slow while joining 2 tables.
Date Mon, 04 May 2015 13:07:56 GMT
hi Olivierspark1.3.1, with java1.8.0.45and add 2 pics .it seems like a GC issue. I also tried
with different parameters like memory size of driver&executor, memory fraction, java opts...but
this issue still happens.

Thanks&amp;Best regards!
罗辉 San.Luo

----- 原始邮件 -----
发件人:Olivier Girardot <>
收件人, user <>
主题:Re: sparksql running slow while joining 2 tables.
日期:2015年05月04日 20点46分

What is you Spark version ?

Le lun. 4 mai 2015 à 11:03, <> a écrit :
hi guys        when i am running a sql  like "select,a.startpoint,a.endpoint, a.piece
from db a join sample b on ( = where (b.startpoint > a.startpoint + 25);"
I found sparksql running slow in minutes which may caused by very long GC and shuffle time.
       table db is created from a txt file size at 56mb while table sample sized at 26mb,
both at small size.       my spark cluster is a standalone  pseudo-distributed spark cluster
with 8g executor and 4g driver manager.       any advises? thank you guys. 
Thanks&amp;Best regards!
罗辉 San.Luo


To unsubscribe, e-mail:

For additional commands, e-mail:

View raw message