spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Younes Naguib <Younes.Nag...@tritondigital.com>
Subject RE: Dynamic partition pruning
Date Mon, 19 Oct 2015 18:33:53 GMT
Done: SPARK-11150

Thanks

From: Xiao Li [mailto:gatorsmile@gmail.com]
Sent: October-16-15 8:21 PM
To: Younes Naguib
Cc: Michael Armbrust; user@spark.apache.org
Subject: Re: Dynamic partition pruning

Hi, Younes,

Maybe you can open a JIRA?

Thanks,

Xiao Li

2015-10-16 12:43 GMT-07:00 Younes Naguib <Younes.Naguib@tritondigital.com<mailto:Younes.Naguib@tritondigital.com>>:
Thanks,
Do you have a Jira I can follow for this?

y

From: Michael Armbrust [mailto:michael@databricks.com<mailto:michael@databricks.com>]
Sent: October-16-15 2:18 PM
To: Younes Naguib
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Dynamic partition pruning

We don't support dynamic partition pruning yet.

On Fri, Oct 16, 2015 at 10:20 AM, Younes Naguib <Younes.Naguib@tritondigital.com<mailto:Younes.Naguib@tritondigital.com>>
wrote:
Hi all

I’m running sqls on spark 1.5.1 and using tables based on parquets.
My tables are not pruned when joined on partition columns.
Ex:
Select .... from tab where partcol=1 will prune on value 1
Select .... from tab join dim on (dim.partcol=tab.partcol) where dim.partcol=1 will scan all
partitions.

Any ideas or workarounds?


Thanks,
Younes



Mime
View raw message