impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Internal Jenkins (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL
Date Sun, 06 Nov 2016 06:50:48 GMT
Internal Jenkins has submitted this change and it was merged.

Change subject: IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL
......................................................................


IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL

This patch adds the ability for the test infrastructure's
Impala-to-PostgreSQL data migration tool to recognize whether the Impala
source tables have primary keys, and if so, CREATE the tables in
PostgreSQL with the same primary keys. This is needed especially for
performing CRUD operations by the random query generator for comparison
with Impala/Kudu tables and equivalent PostgreSQL tables.

I modified the make_create_table_sql() implementation to check the
"universal" Python object model of the table's columns. We generate
CREATE TABLE statements with, or without, a PRIMARY KEY clause. For
Impala-side tables that this tool may create, we also ensure that we
only write such a clause when the table's format supports primary keys
(currently Kudu).

When the random query generator runs, it needs to know that the tables
it's examining in both databases are equivalent. It does this by
examining the tables' names, column names, and column types. I have
added whether the column is a primary key as part of this equivalence
test.

Testing:
- The patch includes some unit and system tests for the tool.
- Actually migrated a few small Kudu and HDFS tables from Impala into
  both PostgreSQL 9.3 and 9.5 and examined the tables in PostgreSQL to
  make sure they had primary keys (or not) as expected.
- Very short discrepancy_searcher.py --explain-only runs to test positive
  and negative cases of Impala/PostgreSQL equivalency.

Change-Id: I447f022e2dc3d4fc8373b7f388c7875a869921b8
Reviewed-on: http://gerrit.cloudera.org:8080/4951
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Internal Jenkins
---
M tests/comparison/db_connection.py
M tests/comparison/tests/fake_query.py
A tests/comparison/tests/test_cursor.py
3 files changed, 271 insertions(+), 10 deletions(-)

Approvals:
  Taras Bobrovytsky: Looks good to me, approved
  Internal Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/4951
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I447f022e2dc3d4fc8373b7f388c7875a869921b8
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tbobrovytsky@cloudera.com>

Mime
View raw message