spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject question about catalyst and TreeNode
Date Tue, 15 Mar 2016 16:01:44 GMT
i am trying to understand some parts of the catalyst optimizer. but i
struggle with one bigger picture issue:

LogicalPlan extends TreeNode, which makes sense since the optimizations
rely on tree transformations like transformUp and transformDown.

but how can a LogicalPlan be a tree? isnt it really a DAG? if it is
possible to create diamond-like operator dependencies, then assumptions
made in tree transformations could be wrong? for example pushing a limit
operator down into a child sounds safe, but if that same child is also used
by another operator (so it has another parent, no longer a tree) then its
not safe at all.

what am i missing here?
thanks! koert

Mime
View raw message