spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] AngersZhuuuu edited a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore
Date Fri, 17 Jan 2020 02:25:28 GMT
AngersZhuuuu edited a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need
to prune partitions again after pushing down to hive metastore
URL: https://github.com/apache/spark/pull/27232#issuecomment-575435484
 
 
   emmmm, prune partition again is because when call `listPartitionsByFilter` kit can't convert
all spark filter expression to hive metastore filter condition. and will get more partition
then exactly wanted. then prune again will spark's method to keep smallest partitions.
   
   If we can't promise all these case was fixed  in `listPartitionsByFilter`, we may still
need this.  and it's cost is negligible

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message