Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of kwiley@keithwiley.com
 designates 67.18.59.5 as permitted sender)
From: Keith Wiley <kwiley@keithwiley.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: How to import extremely "wide" csv tables
Date: Tue, 13 Mar 2012 10:03:28 -0700
Message-Id: <DB59198F-9F04-495C-820D-5A7154AADE75@keithwiley.com>
To: user@hive.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)

Wrapping hive around existing csv files consists of manually naming and =
typing every column during the creation command.  I have several csv =
tables and some of them have a ton of columns.  I would love a way to =
create hive tables which automatically infers the column types by =
attempting various type conversions or regex matches on the data (say =
the first row).  What would be even cooler is if the first row could =
actually be interpreted differently from the rest of the table...as a =
set of string labels to name the columns while the types could be =
automatically inferred from, say, the *second* row.  These csv files are =
currently of this format, with the first row naming the columns.

Does this make sense?

Now, I'm sure that hive doesn't support this yet -- and I admit it is a =
somewhat esoteric desire on my part -- but I'm curious how others would =
suggest approaching it?  I'm thinking of writing a separate isolated =
program that reads the first two rows of a csv file and dumps a text =
string of column names and types in the correct syntax for a hive =
external table creation statement which I would then copy/paste into =
hive...I was just hoping for a simpler solution.

Thoughts?

Thanks.

=
__________________________________________________________________________=
______
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    =
music.keithwiley.com

"You can scratch an itch, but you can't itch a scratch. Furthermore, an =
itch can
itch but a scratch can't scratch. Finally, a scratch can itch, but an =
itch can't
scratch. All together this implies: He scratched the itch from the =
scratch that
itched but would never itch the scratch from the itch that scratched."
                                           --  Keith Wiley
=
__________________________________________________________________________=
______