spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-15365) Metastore relation should fallback to HDFS size if statistics are not available from table meta data.
Date Wed, 25 May 2016 03:59:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Reynold Xin resolved SPARK-15365.
---------------------------------
       Resolution: Fixed
         Assignee: Parth Brahmbhatt
    Fix Version/s: 2.0.0

> Metastore relation should fallback to HDFS size if statistics are not available from
table meta data.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-15365
>                 URL: https://issues.apache.org/jira/browse/SPARK-15365
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Parth Brahmbhatt
>            Assignee: Parth Brahmbhatt
>             Fix For: 2.0.0
>
>
> Currently if a table is used in join operation we rely on Metastore returned size to
calculate if we can convert the operation to Broadcast join. This optimization only kicks
in for table's that have the statistics available in metastore. Hive generally rolls over
to HDFS if the statistics are not available directly from metastore and this seems like a
reasonable choice to adopt given the optimization benefit of using broadcast joins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message