impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Brown (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL
Date Fri, 04 Nov 2016 18:49:10 GMT
Michael Brown has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/4951

Change subject: IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL
......................................................................

IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL

This patch adds the ability for the test infrastructure's
Impala-to-PostgreSQL data migration tool to recognize whether the Impala
source tables have primary keys, and if so, CREATE the tables in
PostgreSQL with the same primary keys. This is needed especially for
performing CRUD operations by the random query generator for comparison
with Impala/Kudu tables and equivalent PostgreSQL tables.

I modified the make_create_table_sql() implementation to check the
"universal" Python object model of the table's columns. We generate
CREATE TABLE statements with, or without, a PRIMARY KEY clause. For
Impala-side tables that this tool may create, we also ensure that we
only write such a clause when the table's format supports primary keys
(currently Kudu).

When the random query generator runs, it needs to know that the tables
it's examining in both databases are equivalent. It does this by
examining the tables' names, column names, and column types. I have
added whether the column is a primary key as part of this equivalence
test.

Testing:
- The patch includes some unit and system tests for the tool.
- Actually migrated a few small Kudu and HDFS tables from Impala into
  both PostgreSQL 9.3 and 9.5 and examined the tables in PostgreSQL to
  make sure they had primary keys (or not) as expected.
- Very short discrepancy_searcher.py --explain-only runs to test positive
  and negative cases of Impala/PostgreSQL equivalency.

Change-Id: I447f022e2dc3d4fc8373b7f388c7875a869921b8
---
M tests/comparison/db_connection.py
M tests/comparison/tests/fake_query.py
A tests/comparison/tests/test_cursor.py
3 files changed, 268 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/4951/2
-- 
To view, visit http://gerrit.cloudera.org:8080/4951
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I447f022e2dc3d4fc8373b7f388c7875a869921b8
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Brown <mikeb@cloudera.com>

Mime
View raw message