spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <>
Subject [GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...
Date Sat, 04 Aug 2018 08:13:27 GMT
Github user cloud-fan commented on the issue:
    > Regarding the second point, this is not an ordinary range join, but an equi-join
with a secondary range condition.
    "an equi-join with a secondary range condition" is a restriction isn't it? IIRC the logical
rewrite approach can work for range join without equi-join, which has broader use cases. e.g.
turning a cartisian join into hash join/SMJ.
    I took a quick look at this patch, it's really bulky, and not safe to be turned on by
default without the spill ability. And more importantly, it can't work for range join without
equi-condition, which I think should be a common use case.
    Can you investigate the logical rewrite approach? I'll attach some related papers if I
find them.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message