Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: dhalperi@apache.org
To: commits@beam.apache.org
Date: Thu, 23 Mar 2017 17:33:04 -0000
Message-Id: <926bf4a1408d42dabe7bf31d1fc9e157@git.apache.org>
Subject: [1/2] beam git commit: [BEAM-1780] BigtableIO: better handling of bad
 split requests
archived-at: Thu, 23 Mar 2017 17:33:06 -0000

Repository: beam
Updated Branches:
  refs/heads/master 82b7b8613 -> c48fa271b


[BEAM-1780] BigtableIO: better handling of bad split requests

The contract for `splitIntoFraction` is that it should only throw if the
reader is in an unknown, bad state. The proper way to reject invalid or
unsatisfiable split requests is to return null.

However, `BigtableIO.Read` will currently throw for simply invalid input
it should reject. This can lead to less effective dynamic work
rebalancing and even stuck jobs.

Related to, but probably not a complete solution for, BEAM-1751.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ba2b76ae
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ba2b76ae
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ba2b76ae

Branch: refs/heads/master
Commit: ba2b76aef40cd184c78a2f10b24718d0b51671b9
Parents: 82b7b86
Author: Dan Halperin <dhalperi@google.com>
Authored: Wed Mar 22 10:21:33 2017 -0700
Committer: Dan Halperin <dhalperi@google.com>
Committed: Thu Mar 23 10:32:53 2017 -0700

----------------------------------------------------------------------
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java    | 21 ++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/beam/blob/ba2b76ae/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
----------------------------------------------------------------------
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 2a8de82..9d02f65 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -998,19 +998,32 @@ public class BigtableIO {
     }
 
     @Override
+    @Nullable
     public final synchronized BigtableSource splitAtFraction(double fraction) {
       ByteKey splitKey;
       try {
         splitKey = rangeTracker.getRange().interpolateKey(fraction);
-      } catch (IllegalArgumentException e) {
+      } catch (RuntimeException e) {
         LOG.info(
-            "%s: Failed to interpolate key for fraction %s.", rangeTracker.getRange(), fraction);
+            "%s: Failed to interpolate key for fraction %s.", rangeTracker.getRange(), fraction, e);
         return null;
       }
       LOG.debug(
           "Proposing to split {} at fraction {} (key {})", rangeTracker, fraction, splitKey);
-      BigtableSource primary = source.withEndKey(splitKey);
-      BigtableSource residual = source.withStartKey(splitKey);
+      BigtableSource primary;
+      BigtableSource residual;
+      try {
+         primary = source.withEndKey(splitKey);
+         residual =  source.withStartKey(splitKey);
+      } catch (RuntimeException e) {
+        LOG.info(
+            "%s: Interpolating for fraction %s yielded invalid split key %s.",
+            rangeTracker.getRange(),
+            fraction,
+            splitKey,
+            e);
+        return null;
+      }
       if (!rangeTracker.trySplitAtPosition(splitKey)) {
         return null;
       }