hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Namit Jain <>
Subject Re: Selecting data based on the clustered columns
Date Fri, 17 Jul 2009 05:42:25 GMT
I am not sure if they are handling this. Let me talk to Prasad offline and get back to you.

On 7/16/09 9:49 PM, "Deepak A" <> wrote:

Hi Namit,

I checked JIRA for any existing tickets on this and figured out that there are plans to support
indexing on queries. This is being discussed at

Can you please check if what we are discussing makes sense in this content or if it is orthogonal
to this.


On Thu, Jul 16, 2009 at 10:26 PM, Deepak A <> wrote:
Hi Namit,

Thanks a lot on the update.
Will do that for sure.


On Thu, Jul 16, 2009 at 7:49 PM, Namit Jain <> wrote:
Right now, bucketing information is not used in a lot of places - it is only used in sampling.
For eg:

If your query was:

Select .. From Posts(tablesample 1 out of 256) a;

Then only the first bucket will be scanned.

Your query can be optimized, but currently it is not. Can you file a jira on that ?
It will help us prioritize this.


On 7/16/09 3:25 AM, "Deepak A" <> wrote:


I have the following table in Hive
Posts(Id, UserId, PostDate, ...) CLUSTERED BY (UserId) SORTED BY (PostDate) INTO 256 BUCKETS;

Since the data is hash partitioned based on the 'UserId' column, buckets were created based
on the hash value of 'UserId'.

Now, when I issue a Select query to fetch all the posts by a particular 'UserId ' (say, Select
count(Id) from Posts where UserId=1), does it scan only the bucket to which 'UserId' is hashed
to?. But, when I run this query, I could see all the buckets being searched for the UserId.

Moreover, I see that's there is a way to sample the table based on the buckets. Why can't
hive automatically figure out the bucket to which UserId is hashed to and search only in that

Can someone clarify me on this?


View raw message