impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jbap...@apache.org
Subject [1/2] incubator-impala git commit: IMPALA-4461: Make sure data gets loaded for wide hbase tables.
Date Fri, 11 Nov 2016 03:45:41 GMT
Repository: incubator-impala
Updated Branches:
  refs/heads/master 0aeb68050 -> c68734ad6


IMPALA-4461: Make sure data gets loaded for wide hbase tables.

Ths patch reverts a change that broke the exhaustive suite of Impala
tests. The change was introduced here:

https://github.com/apache/incubator-impala/commit/ce4c5f67433ebe050261710920972621d625c81c

The orginal problem was that data load was failing when run against a
remote cluster, due to a 4000 byte max for SERDEPROPERTIES.PARAM_VALUE,
a limitation that is well described in HIVE-1364. Locally, when we load
data, we work around the issue here:

https://github.com/apache/incubator-impala/blob/master/bin/create-test-configuration.sh#L99

When testing on CDH remote cluster however, this "fix" never gets applied.
(It also assumes the database will always by postgres.)

I made this change without realizing its full effect, or appreciating
exactly how exhaustive our exhaustive test suite really is. Another
solution will need to be found for the case of remote cluster testing,
but this should unblock the local build for now.

As far as testing, I ran the full suite of tests in query_test/
test_scanners.py, and they all pass after removing these lines.

Change-Id: If2148d6546789c6c53c8e045717081b24ce76689
Reviewed-on: http://gerrit.cloudera.org:8080/5033
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins


Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/b14f3197
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/b14f3197
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/b14f3197

Branch: refs/heads/master
Commit: b14f319708943b14680c77b2afbc502a056d06a1
Parents: 0aeb680
Author: David Knupp <dknupp@cloudera.com>
Authored: Wed Nov 9 17:55:46 2016 -0800
Committer: Internal Jenkins <cloudera-hudson@gerrit.cloudera.org>
Committed: Fri Nov 11 00:37:59 2016 +0000

----------------------------------------------------------------------
 testdata/datasets/functional/schema_constraints.csv | 5 -----
 1 file changed, 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/b14f3197/testdata/datasets/functional/schema_constraints.csv
----------------------------------------------------------------------
diff --git a/testdata/datasets/functional/schema_constraints.csv b/testdata/datasets/functional/schema_constraints.csv
index 790dba4..d6d1111 100644
--- a/testdata/datasets/functional/schema_constraints.csv
+++ b/testdata/datasets/functional/schema_constraints.csv
@@ -118,11 +118,6 @@ table_name:overflow, constraint:restrict_to, table_format:text/none/none
 # seem to like this.
 table_name:widerow, constraint:exclude, table_format:hbase/none/none
 
-# Wide tables fail due to the SERDEPROPERTIES limits. See HIVE-1364.
-table_name:widetable_250_cols, constraints:exclude, table_format:hbase/none/none
-table_name:widetable_500_cols, constraints:exclude, table_format:hbase/none/none
-table_name:widetable_1000_cols, constraints:exclude, table_format:hbase/none/none
-
 # nullformat_custom is used in null-insert tests, which user insert overwrite,
 # which is not supported in hbase. The schema is also specified in HIVE_CREATE
 # with no corresponding LOAD statement.


Mime
View raw message