hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Grover <mgro...@oanda.com>
Subject Re: How to Decide How Many Buckets to Use?
Date Wed, 11 Jul 2012 13:18:29 GMT
Hi Xun,
Here is an answer I had given earlier on the mailing list. See if that helps.

http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3C350967547.114894.1335230199385.JavaMail.root@sms-zimbra-message-store-03.sms.scalar.ca%3E

Mark

----- Original Message -----
From: "Xun TANG" <tangxun.alice@gmail.com>
To: user@hive.apache.org
Sent: Tuesday, July 10, 2012 10:08:22 PM
Subject: How to Decide How Many Buckets to Use?

Hi, 


We use Hive to store user click logs and dig out interesting trends, doing a lot of big table
joins. 
We mostly partition by date, and inside each partition data is bucketed by some hashed-id.



We are wondering how many buckets to use? 
If I understand correctly, Hive DB stores each bucket in one file. One file could be break
down to several buckets based on bucket size. 
Is this one of the criteria to determine bucket size s.t. each file in one block? What else
should be considered? 


Is there a rule-of-thumb in selecting how many buckets to use? Any insight/comment welcome!



Thanks, 
Xun 

Mime
View raw message