Return-Path: 
 <user-return-10230-apmail-hive-user-archive=hive.apache.org@hive.apache.org>
X-Original-To: apmail-hive-user-archive@www.apache.org
Delivered-To: apmail-hive-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27893D46A
	for <apmail-hive-user-archive@www.apache.org>;
 Fri,  7 Sep 2012 14:42:16 +0000 (UTC)
Received: (qmail 37320 invoked by uid 500); 7 Sep 2012 14:42:14 -0000
Delivered-To: apmail-hive-user-archive@hive.apache.org
Received: (qmail 37256 invoked by uid 500); 7 Sep 2012 14:42:14 -0000
Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hive.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hive.apache.org>
List-Post: <mailto:user@hive.apache.org>
List-Id: <user.hive.apache.org>
Reply-To: user@hive.apache.org
Delivered-To: mailing list user@hive.apache.org
Received: (qmail 37248 invoked by uid 99); 7 Sep 2012 14:42:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Sep 2012 14:42:14 +0000
X-ASF-Spam-Status: No, hits=1.8 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sandeepreddy.3647@gmail.com
 designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-wg0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Sep 2012 14:42:08 +0000
Received: by wgbfm10 with SMTP id fm10so379616wgb.5
        for <user@hive.apache.org>; Fri, 07 Sep 2012 07:41:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        bh=eZY/Gcx8tdkKPd2aGoQ59NoKSvpTGw7PqEMuZG3qBPs=;
        b=lrGTor3KtmkMwTqEBC0o1NmY8N2fTHHpiSzGOFQ1hk+l6bhM7xWazcKSBp9st01nFe
         7TPHw6bhvj2ZBedzS6oMtXOd2ty+L7pSygzKQ7HVsn9vk4D3XJjGoGop1nYFDkN/4wEC
         I60E3bkQqaysydRp5YQPot9AjFb1YfzA9NpesZ1N0lGpDPSkMh611w8UzdZJMCJ5GlAF
         gkQIh8BpdRtTsMHTde4taJFgvIsK4vXL2YI9WgkFLVh5d7qXwmipFnH/PmBPO855GoKy
         9qhPmpS7ct8PyTS03uPDRD4kZSseN9I3khvkRqjvXg0Pk//fvsR2yXaUyO+kXy9WQCe+
         WvAw==
MIME-Version: 1.0
Received: by 10.180.20.11 with SMTP id j11mr12766827wie.12.1347028907923; Fri,
 07 Sep 2012 07:41:47 -0700 (PDT)
Received: by 10.194.18.242 with HTTP; Fri, 7 Sep 2012 07:41:47 -0700 (PDT)
In-Reply-To: <9D8A350A3269554E91B45801B5E8CDAC67CC25@SOM-EXCH02.nuance.com>
References: 
 <CAP8D2NYveBp9c3OHLwV3NxWB+Kzf_mFmK=yC9FY82AdsO+ZJXQ@mail.gmail.com>
	<9D8A350A3269554E91B45801B5E8CDAC67CC25@SOM-EXCH02.nuance.com>
Date: Fri, 7 Sep 2012 10:41:47 -0400
Message-ID: 
 <CAP8D2NaE1ufqpLQ-9ncYqurXPoxPyRZK6S_zCXabn_LVOQt8nA@mail.gmail.com>
Subject: Re: How to load csv data into HIVE
From: Sandeep Reddy P <sandeepreddy.3647@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=bcaec53d5b098f1b0504c91d9bd4

--bcaec53d5b098f1b0504c91d9bd4
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,
I wrote a shell script to get csv data but when i run that script on a 12GB
csv its taking more time. If i run a python script will that be faster?

On Fri, Sep 7, 2012 at 10:39 AM, Connell, Chuck <Chuck.Connell@nuance.com>w=
rote:

>  How about a Python script that changes it into plain tab-separated text?
> So it would look like this=85****
>
> ** **
>
> 174969274<tab>14-mar-2006<tab>3522876<tab>
> <tab>14-mar-2006<tab>500000308<tab>65<tab>1<newline>
> etc=85****
>
> ** **
>
> Tab-separated with newlines is easy to read and works perfectly on import=
.
> ****
>
> ** **
>
> Chuck Connell****
>
> Nuance R&D Data Team****
>
> Burlington, MA****
>
> 781-565-4611****
>
> ** **
>
> *From:* Sandeep Reddy P [mailto:sandeepreddy.3647@gmail.com]
> *Subject:* How to load csv data into HIVE****
>
> ** **
>
> Hi,
> Here is the sample data
> "174969274","14-mar-2006","****
>
> 3522876","","14-mar-2006","500000308","65","1"|
> "174969275","19-jul-2006","3523154","","19-jul-2006","500000308","65","1"=
|
> "174969276","31-dec-2005","3530333","","31-dec-2005","500000308","65","1"=
|
> "174969277","14-apr-2005","3531470","","14-apr-2005","500000308","65","1"=
|
>
> How to load this kind of data into HIVE?
> I'm using shell script to get rid of double quotes and '|' but its taking
> very long time to work on each csv which are 12GB each. What is the best
> way to do this?****
>
> ** **
>



--=20
Thanks,
sandeep

--bcaec53d5b098f1b0504c91d9bd4
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,<br>I wrote a shell script to get csv data but when i run that script on=
 a 12GB csv its taking more time. If i run a python script will that be fas=
ter?<br><br><div class=3D"gmail_quote">On Fri, Sep 7, 2012 at 10:39 AM, Con=
nell, Chuck <span dir=3D"ltr">&lt;<a href=3D"mailto:Chuck.Connell@nuance.co=
m" target=3D"_blank">Chuck.Connell@nuance.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">





<div link=3D"blue" vlink=3D"purple" lang=3D"EN-US">
<div>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:&quot;Ca=
libri&quot;,&quot;sans-serif&quot;;color:#1f497d">How about a Python script=
 that changes it into plain tab-separated text? So it would look like this=
=85<u></u><u></u></span></p>

<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:&quot;Ca=
libri&quot;,&quot;sans-serif&quot;;color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<p class=3D"MsoNormal">174969274&lt;tab&gt;14-mar-2006&lt;tab&gt;3522876&lt=
;tab&gt; &lt;tab&gt;14-mar-2006&lt;tab&gt;500000308&lt;tab&gt;65&lt;tab&gt;=
1&lt;newline&gt;<br>
etc=85<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,&quot;sans-serif&quot;;color:#1f497d">Tab-separated with newlines=
 is easy to read and works perfectly on import.<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,&quot;sans-serif&quot;;color:#1f497d"><u></u>=A0<u></u></span></p=
>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,&quot;sans-serif&quot;;color:#1f497d">Chuck Connell</span><span s=
tyle=3D"color:#1f497d"><u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,&quot;sans-serif&quot;;color:#1f497d">Nuance R&amp;D Data Team</s=
pan><span style=3D"color:#1f497d"><u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,&quot;sans-serif&quot;;color:#1f497d">Burlington, MA</span><span =
style=3D"color:#1f497d"><u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,&quot;sans-serif&quot;;color:#1f497d"><a href=3D"tel:781-565-4611=
" value=3D"+17815654611" target=3D"_blank">781-565-4611</a><u></u><u></u></=
span></p>

<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:&quot;Ca=
libri&quot;,&quot;sans-serif&quot;;color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<div style=3D"border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in =
0in 0in">
<p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> Sandeep =
Reddy P [mailto:<a href=3D"mailto:sandeepreddy.3647@gmail.com" target=3D"_b=
lank">sandeepreddy.3647@gmail.com</a>]
<br>
<b>Subject:</b> How to load csv data into HIVE<u></u><u></u></span></p>
</div><div><div class=3D"h5">
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">Hi,<br>
Here is the sample data<br>
&quot;174969274&quot;,&quot;14-mar-2006&quot;,&quot;<u></u><u></u></p>
<div>
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt">3522876&quot;,&quot;&=
quot;,&quot;14-mar-2006&quot;,&quot;500000308&quot;,&quot;65&quot;,&quot;1&=
quot;|<br>
&quot;174969275&quot;,&quot;19-jul-2006&quot;,&quot;3523154&quot;,&quot;&qu=
ot;,&quot;19-jul-2006&quot;,&quot;500000308&quot;,&quot;65&quot;,&quot;1&qu=
ot;|<br>
&quot;174969276&quot;,&quot;31-dec-2005&quot;,&quot;3530333&quot;,&quot;&qu=
ot;,&quot;31-dec-2005&quot;,&quot;500000308&quot;,&quot;65&quot;,&quot;1&qu=
ot;|<br>
&quot;174969277&quot;,&quot;14-apr-2005&quot;,&quot;3531470&quot;,&quot;&qu=
ot;,&quot;14-apr-2005&quot;,&quot;500000308&quot;,&quot;65&quot;,&quot;1&qu=
ot;|<br>
<br>
How to load this kind of data into HIVE?<br>
I&#39;m using shell script to get rid of double quotes and &#39;|&#39; but =
its taking very long time to work on each csv which are 12GB each. What is =
the best way to do this?<u></u><u></u></p>
</div>
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><u></u>=A0<u></u></p>
</div></div></div>
</div>

</blockquote></div><br><br clear=3D"all"><br>-- <br>Thanks,<br>sandeep<br><=
br>

--bcaec53d5b098f1b0504c91d9bd4--