kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ale...@apache.org
Subject [kudu] 03/05: KUDU-2662 TestKuduClient.testClientLocation is flaky
Date Mon, 28 Jan 2019 23:58:23 GMT
This is an automated email from the ASF dual-hosted git repository.

alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git

commit 1f1f875cfb6a26b55821edd236f81f28ce06759b
Author: Will Berkeley <wdberkeley@gmail.org>
AuthorDate: Mon Jan 28 14:16:52 2019 -0800

    KUDU-2662 TestKuduClient.testClientLocation is flaky
    
    At least on Linux, multimaster Kudu tests are run with each master on
    its own interface. E.g., in a failed run of testClientLocation, the
    three masters were running on 127.26.241.{124,125,126}. Each client is
    assigned a location by each master it connects to. The trick was, the
    client connection appeared to come from 127.26.241.X for the master on
    127.26.241.X. assign-locations.py remembers assigned locations by IP,
    so it was attempting to assign 3 locations for one client, plus 3 for
    the tablet servers. The test wrongly anticipated 4 assignments and
    configured the script to only do 4 assignments, leading to issues like
    
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) W0126 16:05:46.987164
27624 master_service.cc:535] unable to assign location to client {username='jenkins'}@127.26.241.126:60372:
Runtime error: failed to run location mapping command: Runtime error: /data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py:
process exited with non-zero status 1: Traceback (most recent call last):
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)   File "/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 244, in <module>
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)     main()
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)   File "/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 239, in main
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)     LocationAssignmentRule(args.location_mapping_rules),
args.uid, args.relaxed)
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)   File "/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 198, in get_location
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)     seq + 1, uid,
rule.location_mapping_rules, json.dumps(state)))
    16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) Exception: too
many unique identifiers (5) to assign next location to 127.26.241.126 using mapping rules
['/L0:4']. State: {"mapping_rules": ["/L0:4"], "mappings": {"127.26.241.124": "/L0", "127.26.241.65":
"/L0", "127.26.241.66": "/L0", "127.26.241.67": "/L0"}, "seq": 4}
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) W0126 16:05:47.073348
27688 master_service.cc:535] unable to assign location to client {username='jenkins'}@127.26.241.125:41204:
Runtime error: failed to run location mapping command: Runtime error: /data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py:
process exited with non-zero status 1: Traceback (most recent call last):
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)   File "/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 244, in <module>
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)     main()
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)   File "/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 239, in main
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)     LocationAssignmentRule(args.location_mapping_rules),
args.uid, args.relaxed)
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)   File "/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 198, in get_location
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)     seq + 1, uid,
rule.location_mapping_rules, json.dumps(state)))
    16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) Exception: too
many unique identifiers (5) to assign next location to 127.26.241.125 using mapping rules
['/L0:4']. State: {"mapping_rules": ["/L0:4"], "mappings": {"127.26.241.124": "/L0", "127.26.241.65":
"/L0", "127.26.241.66": "/L0", "127.26.241.67": "/L0"}, "seq": 4}
    
    This didn't cause a test failure in and of itself- the test would still
    pass if the leader master was first to try and assign a location to the
    client. So the test would randomly fail.
    
    The solution is just to up the number of allowed assignments. Since our
    Java tests are very simple and assign a single location only, this will
    avoid the problem.
    
    Change-Id: I2fed3eb62ab9d0f8c1baaa40726355a2a6737b3a
    Reviewed-on: http://gerrit.cloudera.org:8080/12289
    Tested-by: Kudu Jenkins
    Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
---
 .../src/test/java/org/apache/kudu/client/TestKuduClient.java  |  2 +-
 .../src/main/java/org/apache/kudu/test/KuduTestHarness.java   | 11 +++++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
index b849e9f..e11a58e 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
@@ -1168,7 +1168,7 @@ public class TestKuduClient {
 
   @Test(timeout = 100000)
   @LocationConfig(locations = {
-      "/L0:4",
+      "/L0:6", // 3 masters, 1 client, 3 tablet servers: 3 * 1 + 3 = 6.
   })
   public void testClientLocation() throws Exception {
     // Do something that will cause the client to connect to the cluster.
diff --git a/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java
b/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java
index 44f6a22..a462853 100644
--- a/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java
+++ b/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java
@@ -437,6 +437,17 @@ public class KuduTestHarness extends ExternalResource {
    * define a location mapping for the cluster. Location
    * mappings are defined as a series of 'location:number'
    * pairs.
+   *
+   * Note that, in many Kudu tests, multiple masters will be run, each
+   * on their own network interface within the same machine, and client
+   * connections will appear to come from the same interface as the
+   * master being connected to. So, for example, if there are two
+   * clients, three masters, and three tablet servers, nine locations
+   * will be assigned: each client will get a location from each
+   * master (from a different IP), and each tablet server will get a
+   * location. The easiest way to work around this for our simple
+   * Java client tests is to set the number of mappings to be something
+   * at least (# masters) * (# clients) + (# tablet servers)
    */
   @Retention(RetentionPolicy.RUNTIME)
   @Target({ElementType.METHOD})


Mime
View raw message