Mailing-List: contact hive-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hive-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of athusoo@facebook.com designates
 69.63.179.25 as permitted sender)
From: Ashish Thusoo <athusoo@facebook.com>
To: "hive-user@hadoop.apache.org" <hive-user@hadoop.apache.org>
Date: Wed, 26 May 2010 10:17:07 -0700
Subject: RE: Garbage data in metadata store?
Thread-Topic: Garbage data in metadata store?
Thread-Index: Acr8zqORxv8I247fS8eYVR/GtLwfQwAKIPcw
Message-ID: 
 <68B7689C98024D43B4C2709456F0B5200A612F0463@SC-MBXC1.TheFacebook.com>
References: <AANLkTikSWSJ0d4PmKk13U_diJvh9OdzVaWeFJeF6S7LA@mail.gmail.com>
In-Reply-To: <AANLkTikSWSJ0d4PmKk13U_diJvh9OdzVaWeFJeF6S7LA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_68B7689C98024D43B4C2709456F0B5200A612F0463SCMBXC1TheFac_"
MIME-Version: 1.0

--_000_68B7689C98024D43B4C2709456F0B5200A612F0463SCMBXC1TheFac_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Do you have partitions in the table? Storage descriptors can also be associ=
ated with partitions.

Ashish

________________________________
From: Ted Xu [mailto:ted.xu.ml@gmail.com]
Sent: Wednesday, May 26, 2010 5:26 AM
To: hive-user@hadoop.apache.org
Subject: Garbage data in metadata store?

Hi all,

I want to replicate hive metadata to another place, while I found my hive m=
etadata contains a big portion of data looks like garbage.

In my understanding, the hive metadata store use 'Storage Descriptor' to ke=
ep relationship between tables and columns. But the 'SD_ID' columns in tabl=
e 'TBLS' and 'COLUMNS' are unbalanced in count, as shown below:

mysql> select count(distinct SD_ID) from tbls;
+-----------------------+
| count(distinct SD_ID) |
+-----------------------+
|                   764 |
+-----------------------+
1 row in set (0.00 sec)

mysql> select count(distinct SD_ID) from columns;
+-----------------------+
| count(distinct SD_ID) |
+-----------------------+
|                  5219 |
+-----------------------+
1 row in set (0.05 sec)

Is that mean table 'columns' contains garbage data? If so, then how it is g=
enerated?

--
Best Regards,
Ted Xu

--_000_68B7689C98024D43B4C2709456F0B5200A612F0463SCMBXC1TheFac_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dus-ascii">
<META content=3D"MSHTML 6.00.6000.17023" name=3DGENERATOR></HEAD>
<BODY>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D220181617-26052010><FONT face=3DA=
rial=20
color=3D#0000ff size=3D2>Do you have partitions in the table? Storage descr=
iptors=20
can also be associated with partitions.</FONT></SPAN></DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D220181617-26052010><FONT face=3DA=
rial=20
color=3D#0000ff size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D220181617-26052010><FONT face=3DA=
rial=20
color=3D#0000ff size=3D2>Ashish</FONT></SPAN></DIV><BR>
<DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr align=3Dleft>
<HR tabIndex=3D-1>
<FONT face=3DTahoma size=3D2><B>From:</B> Ted Xu [mailto:ted.xu.ml@gmail.co=
m]=20
<BR><B>Sent:</B> Wednesday, May 26, 2010 5:26 AM<BR><B>To:</B>=20
hive-user@hadoop.apache.org<BR><B>Subject:</B> Garbage data in metadata=20
store?<BR></FONT><BR></DIV>
<DIV></DIV>Hi all,
<DIV><BR></DIV>
<DIV>I want to replicate hive metadata to another place, while I found my h=
ive=20
metadata contains a big portion of data looks like garbage.</DIV>
<DIV><BR></DIV>
<DIV>In my understanding, the hive metadata store use 'Storage Descriptor' =
to=20
keep relationship between tables and columns.&nbsp;But the 'SD_ID'&nbsp;col=
umns=20
in table 'TBLS' and 'COLUMNS' are unbalanced in count, as shown below:</DIV=
>
<DIV><BR></DIV>
<DIV>mysql&gt; select count(distinct SD_ID) from tbls;</DIV>
<DIV>+-----------------------+</DIV>
<DIV>| count(distinct SD_ID) |</DIV>
<DIV>+-----------------------+</DIV>
<DIV>| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 764=20
|&nbsp;</DIV>
<DIV>+-----------------------+</DIV>
<DIV>1 row in set (0.00 sec)</DIV>
<DIV><BR></DIV>
<DIV>mysql&gt; select count(distinct SD_ID) from columns;</DIV>
<DIV>+-----------------------+</DIV>
<DIV>| count(distinct SD_ID) |</DIV>
<DIV>+-----------------------+</DIV>
<DIV>| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;5219=20
|&nbsp;</DIV>
<DIV>+-----------------------+</DIV>
<DIV>1 row in set (0.05 sec)</DIV>
<DIV><BR></DIV>
<DIV>Is that mean table 'columns' contains garbage data? If so, then how it=
 is=20
generated?&nbsp;</DIV>
<DIV><BR>-- <BR>Best Regards,<BR>Ted Xu<BR></DIV></BODY></HTML>

--_000_68B7689C98024D43B4C2709456F0B5200A612F0463SCMBXC1TheFac_--