kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From granthe...@apache.org
Subject [kudu] 02/02: [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
Date Fri, 21 Jun 2019 16:04:13 GMT
This is an automated email from the ASF dual-hosted git repository.

granthenke pushed a commit to branch branch-1.10.x
in repository https://gitbox.apache.org/repos/asf/kudu.git

commit 322c3d397d3b5dfc14ca2acc52efc1d0fc50c013
Author: Will Berkeley <wdberkeley@gmail.com>
AuthorDate: Thu Jun 20 14:12:23 2019 -0700

    [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
    
    TestKuduBackup.testRandomBackupAndRestore creates a random table with
    random partitions. In a run with TSAN binaries, I saw the following
    partition schema:
    
    partition_schema {
      hash_bucket_schemas {
        columns {
          name: "int160"
        }
        num_buckets: 8
        seed: 395460932
      }
      hash_bucket_schemas {
        columns {
          name: "int321"
        }
        num_buckets: 6
       seed: 2072428334
      }
      hash_bucket_schemas {
        columns {
          name: "int82"
        }
        num_buckets: 9
        seed: 138096156
      }
    }
    
    So there were 8 * 6 * 9 = 432 tablets being created on a 3-tserver
    minicluster running TSAN. As a result, the test timed out waiting for
    the table to finish creating.
    
    To prevent creating tables with too many tablets, this patch reduces the
    maximum number of hash buckets to 3 and it reduces the maximum nesting of
    hash partitioning to 2. Combined with the existing maximum of 8 range
    partitions, this means no table will consist of more than 9 * 8 = 72
    tablets. Hopefully this is few enough that create table will always
    finish in a timely manner.
    
    Change-Id: Id937f4513bb0ebaf6dec345a141ed9aef2c44388
    Reviewed-on: http://gerrit.cloudera.org:8080/13693
    Reviewed-by: Grant Henke <granthenke@apache.org>
    Tested-by: Kudu Jenkins
    (cherry picked from commit 648b5b85aad0df92ab86ac1a9130d3ffdc7d92d9)
    Reviewed-on: http://gerrit.cloudera.org:8080/13700
    Reviewed-by: Andrew Wong <awong@cloudera.com>
---
 .../src/main/java/org/apache/kudu/util/SchemaGenerator.java           | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java b/java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
index 4116da8..6a97382 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
@@ -235,11 +235,11 @@ public class SchemaGenerator {
     final List<ColumnSchema> keyColumns = schema.getPrimaryKeyColumns();
 
     // Add hash partitioning (Max out at 3 levels to avoid being excessive).
-    int hashPartitionLevels = random.nextInt(Math.min(keyColumns.size(), 3)) + 1;
+    int hashPartitionLevels = random.nextInt(Math.min(keyColumns.size(), 2)) + 1;
     for (int i = 0; i < hashPartitionLevels; i++) {
       final ColumnSchema hashColumn = keyColumns.get(i);
       // TODO(ghenke): Make buckets configurable.
-      final int hashBuckets = random.nextInt(8) + MIN_HASH_BUCKETS;
+      final int hashBuckets = random.nextInt(2) + MIN_HASH_BUCKETS;
       final int hashSeed = random.nextInt();
       options.addHashPartitions(Arrays.asList(hashColumn.getName()), hashBuckets, hashSeed);
     }


Mime
View raw message