spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "梅西0247" <>
Subject Is it possible to turn a SortMergeJoin into BroadcastHashJoin?
Date Mon, 20 Jun 2016 11:06:41 GMT
Hi everyone, 
I ran a SQL join statement on Spark 1.6.1 like this:
select * from table1 a join table2 b on a.c1 = b.c1 where a.c2 < 1000;and it took quite
a long time because It is a SortMergeJoin and the two tables are big.

In fact,  the size of filter result(select * from a where a.c2 < 1000) is very small,
and I think a better solution is to use a BroadcastJoin with the filter result, but  I know 
the physical plan is static and it won't be changed.
So, can we make the physical plan more adaptive? (In this example, I mean using a  BroadcastHashJoin
instead of SortMergeJoin automatically. )

View raw message