Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
	boundary="===============6090740964719781080=="
MIME-Version: 1.0
Subject: Review Request: Add timestamp column with index to the partition
 stats table.
From: "Kevin Wilfong" <kevinwilfong@fb.com>
To: "Ning Zhang" <nzhang@fb.com>, "Yongqiang He" <heyongqiangict@gmail.com>
Date: Tue, 27 Sep 2011 23:58:42 -0000
Message-ID: <20110927235842.9574.73151@reviews.apache.org>
Cc: "Kevin Wilfong" <kevinwilfong@fb.com>,"hive" <dev@hive.apache.org>

--===============6090740964719781080==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2079/
-----------------------------------------------------------

Review request for hive, Yongqiang He and Ning Zhang.


Summary
-------

I added a timestamp column ts to the partition statistics table which defau=
lts to the current_timestamp.  I also added code to create an index on that=
 column, and verify that index exists when we check if the table exists.

I also took the opportunity to fix another problem.  Every time we change t=
he schema of the partition statistics table we give it a slightly different=
 name, like PARTITION_STATS, PARITION_STATISTICS, PARTITION_STAT_TBL, etc. =
 Instead, I want to put a number at the end of the table name, here I have =
PARTITION_STATS_V2, instead of trying to come up on a new variation of name=
, we can just increment the final number, this will also make it easy to id=
entify old tables which can be dropped.

Checking whether the index exists may not be worth the time it takes.  We h=
ave to check this every time we init JDBCStatsPublisher, unless the table d=
oesn't exist, and if it doesn't exist, it's not the end of the world, it ju=
st means any scripts which try to use the index will be slower, and the ind=
ex can always be added later.  Also, the chance the program creates the tab=
le, but is interrupted before it can create the index is low.  I added the =
check because I thought the chance of having to try and find the reason why=
 Hive slowed down, and having to find that a clean up script is running slo=
w, and hence holding the locks for a long time, sounded painful, and hence =
the check would be worth it, but I am open to debate.


This addresses bug HIVE-2471.
    https://issues.apache.org/jira/browse/HIVE-2471


Diffs
-----

  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher=
.java 1175957 =

  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsSetupCons=
tants.java 1175957 =

  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.jav=
a 1175957 =


Diff: https://reviews.apache.org/r/2079/diff


Testing
-------

I ran TestStatsPublisherEnhanced using both derby and MySQL, and verified a=
ll the tests succeeded.

I also ran a few queries and verified that the table and index were created=
 and that the rows, including timestamp, appeared in the table.


Thanks,

Kevin


--===============6090740964719781080==--