Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of qp.wschung@gmail.com designates
 209.85.214.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=iwpj8sI+Ks65rBHVhHdSmfgfSVChMfhUGI972fMbPyfCoenSWm9+bu/FBGVcFLxyt/
         Iv6xtrvcluclqocN0gQtHuv2CJn2raNB6jbs4y4PVud6TM2AJTQi7UFg3TEc6mlBBlHB
         L23rvBeESkpqPIF7G+LZhdR9U4DqebXmokBWQ=
MIME-Version: 1.0
In-Reply-To: <4DF6995B.5090903@gmail.com>
References: <BANLkTinuXVEJzK8HE1fME2kD-EaPEXTJXA@mail.gmail.com>
	<4DF6995B.5090903@gmail.com>
Date: Tue, 14 Jun 2011 11:57:04 -0400
Message-ID: <BANLkTin3tY9WETjHhPmLs2KuEgcneEH_fg@mail.gmail.com>
Subject: Re: load data unit of work
From: W S Chung <qp.wschung@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=005045016c73502eae04a5ae16f1

--005045016c73502eae04a5ae16f1
Content-Type: text/plain; charset=ISO-8859-1

My question is a "what if" question, not a production issue. It seems
natural, when replacing traditional database with hive, to ask
how much robustness is sacrificed for scalability. My concern is that if a
file is partially loaded, there might not be an easy way to clean up the
already loaded data before re-loading the data. The lack of unique index
also does not make it easy to avoid duplicate data either, although
duplicated data can perhaps be deleted after the load.

On Mon, Jun 13, 2011 at 7:12 PM, Martin Konicek <martin.konicek@gmail.com>wrote:

> Hi,
>
> I think this is a problem with open source in general and sometimes it can
> be very frustrating.
> However, your question is more of a "what if" question - you're not in the
> trouble of finding a horrible bug after you deployed to production, am I
> right?
>
> Regarding your question, I would guess that if LOAD DATA INPATH crashes
> while moving files into the Hive warehouse, the data which was moved will
> appear as legitimate loaded data. Or the files will be moved but the
> metadata will not be updated. In any case, you should detect the crash and
> redo the operation. The easiest answer might actually be to look into the
> source code - sometimes it can be easier to find than one would expect.
>
> Not a complete answer, but hope this helps a bit.
>
> Martin
>
>
> On 14/06/2011 00:47, W S Chung wrote:
>
>> I submit a question like this before, but somehow that question is never
>> delivered. I can even find my question in google. Since I cannot find any
>> admin e-mail/feedback form on the hive website that I can ask why the last
>> question is not delivered. There is not much option other than to post the
>> question again and hope that the question get through this time. Sorry for
>> the double posting if you have seen my last e-mail.
>>
>> What is the behaviour if  a client of hive crashes in the middle of
>> running a "load data inpath" for either a local file or a file on HDFS? Will
>> the file be partially loaded in the db? Thanks.
>>
>>
>>

--005045016c73502eae04a5ae16f1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

My question is a &quot;what if&quot; question, not a production issue. It s=
eems natural, when replacing traditional database with hive, to ask<br>how =
much robustness is sacrificed for scalability. My concern is that if a file=
 is partially loaded, there might not be an easy way to clean up the alread=
y loaded data before re-loading the data. The lack of unique index also doe=
s not make it easy to avoid duplicate data either, although duplicated data=
 can perhaps be deleted after the load.<br>
<br><div class=3D"gmail_quote">On Mon, Jun 13, 2011 at 7:12 PM, Martin Koni=
cek <span dir=3D"ltr">&lt;<a href=3D"mailto:martin.konicek@gmail.com">marti=
n.konicek@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quo=
te" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;=
">
Hi,<br>
<br>
I think this is a problem with open source in general and sometimes it can =
be very frustrating.<br>
However, your question is more of a &quot;what if&quot; question - you&#39;=
re not in the trouble of finding a horrible bug after you deployed to produ=
ction, am I right?<br>
<br>
Regarding your question, I would guess that if LOAD DATA INPATH crashes whi=
le moving files into the Hive warehouse, the data which was moved will appe=
ar as legitimate loaded data. Or the files will be moved but the metadata w=
ill not be updated. In any case, you should detect the crash and redo the o=
peration. The easiest answer might actually be to look into the source code=
 - sometimes it can be easier to find than one would expect.<br>

<br>
Not a complete answer, but hope this helps a bit.<br><font color=3D"#888888=
">
<br>
Martin</font><div><div></div><div class=3D"h5"><br>
<br>
On 14/06/2011 00:47, W S Chung wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
I submit a question like this before, but somehow that question is never de=
livered. I can even find my question in google. Since I cannot find any adm=
in e-mail/feedback form on the hive website that I can ask why the last que=
stion is not delivered. There is not much option other than to post the que=
stion again and hope that the question get through this time. Sorry for the=
 double posting if you have seen my last e-mail.<br>

<br>
What is the behaviour if =A0a client of hive crashes in the middle of runni=
ng a &quot;load data inpath&quot; for either a local file or a file on HDFS=
? Will the file be partially loaded in the db? Thanks.<br>
<br>
<br>
</blockquote>
</div></div></blockquote></div><br>

--005045016c73502eae04a5ae16f1--