hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ameet chaubal <>
Subject Re: millions of columns in Hive
Date Thu, 22 Dec 2011 20:55:28 GMT
thanks Kirk,
Actually, it saves the columns in a row format in a table "COLUMNS"
So all the columns are stored as records in this table. i.e. for a million row table, there
will be a million rows which MySQL should have no problem with, right?

I did see a post regarding the 4000 bytes,but that was for serde definition to map an HBase
table with a large number of columns, which required that the serde string would contain a
large number of characters.

However, in this case, since the columns would be stored as rows, it ought to create a million
rows for this table... I hope ... 




 From: Kirk True <>
Cc: ameet chaubal <> 
Sent: Thursday, December 22, 2011 1:49 PM
Subject: Re: millions of columns in Hive

IIRC, the names and types of the columns are kept in a *_PARAMS table in the metastore. At
one point (depending on your DB) this was only ~4000 characters and thus posed a limitation
on the number of columns one could use.

I believe there was work to turn some of the *_PARAMS table columns
    into CLOBs (or equivalent).


On 12/20/11 11:08 AM, ameet chaubal wrote: 
Hi all,
>is there a limitation in Hive to create a large number of columns, say a few million or
so? If one is using MySQL for metastore, where would this limitation come from, if so? 
View raw message