Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1244.3)
Subject: Re: How to load lines into Hive while breaking them by words?
From: Adriaan Tijsseling <adriaan@tijsseling.com>
In-Reply-To: 
 <CANYdkkPdN=LtpQYavFkdixmMpVAJanfAvH9mfU1PaM3i9mJBHw@mail.gmail.com>
Date: Tue, 27 Sep 2011 09:20:53 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <75CDA121-B28B-4DB3-811F-067C3A2137C7@tijsseling.com>
References: 
 <CANYdkkPdN=LtpQYavFkdixmMpVAJanfAvH9mfU1PaM3i9mJBHw@mail.gmail.com>
To: user@hive.apache.org

Use a regexserde to split the text in words. There's documentation on =
the hive wiki.=20
But it might be better to use a script. See the post by Shouguo Li =
earlier on this mailing list.
After all, when you use a Python script, for example, you could use the =
Natural Language Processing Toolkit to get much better function that =
splits text into a list of words.

Adriaan

On 2011/09/27, at 05:40, Mark Kerzner wrote:

> Hi,
>=20
> a simple question - if I have a book as a text, and I want to load it =
into a
> Hive table, with one word forming one entry, how should I do it?
>=20
> Thank you,
> Mark