spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhenhua Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node
Date Mon, 27 Mar 2017 03:26:41 GMT
Zhenhua Wang created SPARK-20104:
------------------------------------

             Summary: Don't estimate IsNull or IsNotNull predicates for non-leaf node
                 Key: SPARK-20104
                 URL: https://issues.apache.org/jira/browse/SPARK-20104
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Zhenhua Wang


In current stage, we don't have advanced statistics such as sketches or histograms. As a result,
some operator can't estimate `nullCount` accurately. E.g. left outer join estimation does
not accurately update `nullCount` currently. So for IsNull and IsNotNull predicates, we only
estimate them when the child is a leaf node, whose `nullCount` is accurate. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message