kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject kudu git commit: docs: KUDU-1767. Document possible client op reordering
Date Thu, 12 Jan 2017 20:47:58 GMT
Repository: kudu
Updated Branches:
  refs/heads/branch-1.2.x a23e9d645 -> 97ad14d8a


docs: KUDU-1767. Document possible client op reordering

This patch documents the possibility of client reordering when using
concurrent flush modes in both the C++ and Java clients.

Also adds a note to the "Transaction Semantics" section of the user
guide.

Tested the docs rendering by building the user guide and the C++ and
Java API docs locally.

Also updated some old issue URLs to point to the ASF JIRA instance.

Change-Id: I65215d833c65e54fcf080d61adc5f6ed3d303224
Reviewed-on: http://gerrit.cloudera.org:8080/5464
Reviewed-by: Todd Lipcon <todd@apache.org>
Tested-by: Todd Lipcon <todd@apache.org>
(cherry picked from commit eb039566cb5a7e14d68e3c2e207c2b5187044522)
Reviewed-on: http://gerrit.cloudera.org:8080/5699


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/97ad14d8
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/97ad14d8
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/97ad14d8

Branch: refs/heads/branch-1.2.x
Commit: 97ad14d8a88e5f5a9505969f73a9b7938da47269
Parents: a23e9d6
Author: Mike Percy <mpercy@apache.org>
Authored: Sat Dec 10 18:23:47 2016 +0000
Committer: Todd Lipcon <todd@apache.org>
Committed: Thu Jan 12 20:47:45 2017 +0000

----------------------------------------------------------------------
 docs/transaction_semantics.adoc                 |  9 ++-
 .../apache/kudu/client/AsyncKuduSession.java    | 77 +++++++++++++-------
 .../org/apache/kudu/client/KuduSession.java     | 25 +++++--
 .../kudu/client/SessionConfiguration.java       | 61 ++++++++++------
 src/kudu/client/client.h                        | 17 +++++
 5 files changed, 132 insertions(+), 57 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/97ad14d8/docs/transaction_semantics.adoc
----------------------------------------------------------------------
diff --git a/docs/transaction_semantics.adoc b/docs/transaction_semantics.adoc
index 3cf0785..a070634 100644
--- a/docs/transaction_semantics.adoc
+++ b/docs/transaction_semantics.adoc
@@ -223,7 +223,14 @@ in some situations, at the moment. Below are the details and next, some
recommen
   See <<recommendations>> for a workaround.
 * Impala scans are currently performed as `READ_LATEST` and have no consistency
   guarantees.
-
+* In `AUTO_BACKGROUND_FLUSH` mode, or when using "async" flushing mechanisms,
+  writes applied to a single client session may become reordered due to the
+  concurrency of flushing the data to the server. This may be particularly
+  noticeable if a single row is quickly updated with different values in
+  succession. This phenomenon affects all client API implementations.
+  Workarounds are described in the API documentation for the respective
+  implementations in the docs for `FlushMode` or `AsyncKuduSession`.
+  See link:https://issues.apache.org/jira/browse/KUDU-1767[KUDU-1767].
 
 [[recommendations]]
 === Recommendations

http://git-wip-us.apache.org/repos/asf/kudu/blob/97ad14d8/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
index ac847fc..f8a2115 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
@@ -49,46 +49,64 @@ import org.apache.kudu.util.AsyncUtil;
 import org.apache.kudu.util.Slice;
 
 /**
- * A AsyncKuduSession belongs to a specific AsyncKuduClient, and represents a context in
- * which all read/write data access should take place. Within a session,
+ * An {@code AsyncKuduSession} belongs to a specific {@link AsyncKuduClient}, and represents
a
+ * context in which all write data access should take place. Within a session,
  * multiple operations may be accumulated and batched together for better
  * efficiency. Settings like timeouts, priorities, and trace IDs are also set
- * per session.<p>
+ * per session.
  *
- * AsyncKuduSession is separate from KuduTable because a given batch or transaction
- * may span multiple tables. This is particularly important in the future when
- * we add ACID support, but even in the context of batching, we may be able to
- * coalesce writes to different tables hosted on the same server into the same
- * RPC.<p>
- *
- * AsyncKuduSession is separate from AsyncKuduClient because, in a multi-threaded
+ * <p>{@code AsyncKuduSession} is separate from {@link AsyncKuduClient} because, in
a multi-threaded
  * application, different threads may need to concurrently execute
  * transactions. Similar to a JDBC "session", transaction boundaries will be
  * delineated on a per-session basis -- in between a "BeginTransaction" and
  * "Commit" call on a given session, all operations will be part of the same
- * transaction. Meanwhile another concurrent Session object can safely run
- * non-transactional work or other transactions without interfering.<p>
+ * transaction. Meanwhile another concurrent session object can safely run
+ * non-transactional work or other transactions without interfering.
  *
- * Therefore, this class is <b>not</b> thread-safe.<p>
+ * <p>Therefore, this class is <b>not</b> thread-safe.
  *
- * Additionally, there is a guarantee that writes from different sessions do not
+ * <p>Additionally, there is a guarantee that writes from different sessions do not
  * get batched together into the same RPCs -- this means that latency-sensitive
- * clients can run through the same AsyncKuduClient object as throughput-oriented
+ * clients can run through the same {@link AsyncKuduClient} object as throughput-oriented
  * clients, perhaps by setting the latency-sensitive session's timeouts low and
  * priorities high. Without the separation of batches, a latency-sensitive
  * single-row insert might get batched along with 10MB worth of inserts from the
- * batch writer, thus delaying the response significantly.<p>
+ * batch writer, thus delaying the response significantly.
+ *
+ * <p>Timeouts are handled differently depending on the flush mode.
+ * With {@link SessionConfiguration.FlushMode#AUTO_FLUSH_SYNC AUTO_FLUSH_SYNC}, the timeout
is set
+ * on each {@linkplain #apply apply}()'d operation.
+ * With {@link SessionConfiguration.FlushMode#AUTO_FLUSH_BACKGROUND AUTO_FLUSH_BACKGROUND}
and
+ * {@link SessionConfiguration.FlushMode#MANUAL_FLUSH MANUAL_FLUSH}, the timeout is assigned
to a
+ * whole batch of operations upon {@linkplain #flush flush}()'ing. It means that in a situation
+ * with a timeout of 500ms and a flush interval of 1000ms, an operation can be outstanding
for up to
+ * 1500ms before being timed out.
+ *
+ * <p><strong>Warning: a note on out-of-order operations</strong>
+ *
+ * <p>When using {@code AsyncKuduSession}, it is not difficult to trigger concurrent
flushes on
+ * the same session. The result is that operations applied in a particular order within a
single
+ * session may be applied in a different order on the server side, even for a single tablet.
To
+ * prevent this behavior, ensure that only one flush is outstanding at a given time (the
maximum
+ * concurrent flushes per {@code AsyncKuduSession} is hard-coded to 2).
+ *
+ * <p>If operation interleaving would be unacceptable for your application, consider
using one of
+ * the following strategies to avoid it:
  *
- * Though we currently do not have transactional support, users will be forced
- * to use a AsyncKuduSession to instantiate reads as well as writes.  This will make
- * it more straight-forward to add RW transactions in the future without
- * significant modifications to the API.<p>
+ * <ol>
+ * <li>When using {@link SessionConfiguration.FlushMode#MANUAL_FLUSH MANUAL_FLUSH}
mode,
+ * wait for one {@link #flush flush()} to {@code join()} before triggering another flush.
+ * <li>When using {@link SessionConfiguration.FlushMode#AUTO_FLUSH_SYNC AUTO_FLUSH_SYNC}
+ * mode, wait for each {@link #apply apply()} to {@code join()} before applying another operation.
+ * <li>Consider not using
+ * {@link SessionConfiguration.FlushMode#AUTO_FLUSH_BACKGROUND AUTO_FLUSH_BACKGROUND} mode.
+ * <li>Make your application resilient to out-of-order application of writes.
+ * <li>Avoid applying an {@link Operation} on a particular row until any previous write
to that
+ * row has been successfully flushed.
+ * </ol>
  *
- * Timeouts are handled differently depending on the flush mode.
- * With AUTO_FLUSH_SYNC, the timeout is set on each apply()'d operation.
- * With AUTO_FLUSH_BACKGROUND and MANUAL_FLUSH, the timeout is assigned to a whole batch
of
- * operations upon flush()'ing. It means that in a situation with a timeout of 500ms and
a flush
- * interval of 1000ms, an operation can be outstanding for up to 1500ms before being timed
out.
+ * <p>For more information on per-session operation interleaving, see
+ * <a href="https://issues.apache.org/jira/browse/KUDU-1767">KUDU-1767</a>.
  */
 @InterfaceAudience.Public
 @InterfaceStability.Unstable
@@ -489,12 +507,15 @@ public class AsyncKuduSession implements SessionConfiguration {
 
   /**
    * Apply the given operation.
-   * The behavior of this function depends on the current flush mode. Regardless
-   * of flush mode, however, Apply may begin to perform processing in the background
-   * for the call (e.g looking up the tablet, etc).
+   * <p>
+   * The behavior of this method depends on the configured
+   * {@link SessionConfiguration.FlushMode FlushMode}. Regardless
+   * of flush mode, however, {@code apply()} may begin to perform processing in the background
+   * for the call (e.g looking up the tablet location, etc).
    * @param operation operation to apply
    * @return a Deferred to track this operation
    * @throws KuduException if an error happens or {@link PleaseThrottleException} is triggered
+   * @see SessionConfiguration.FlushMode FlushMode
    */
   public Deferred<OperationResponse> apply(final Operation operation) throws KuduException
{
     Preconditions.checkNotNull(operation, "Can not apply a null operation");

http://git-wip-us.apache.org/repos/asf/kudu/blob/97ad14d8/java/kudu-client/src/main/java/org/apache/kudu/client/KuduSession.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduSession.java b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduSession.java
index 0c226ee..4e68f44 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduSession.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduSession.java
@@ -45,24 +45,35 @@ public class KuduSession implements SessionConfiguration {
   }
 
   /**
-   * Blocking call with a different behavior based on the flush mode. PleaseThrottleException
is
-   * managed by this method and will not be thrown, unlike {@link AsyncKuduSession#apply}.
+   * Apply a given {@link Operation} to Kudu as part of this session.
+   *
    * <p>
+   * This is a blocking call that has different behavior based on the configured flush mode:
+   *
    * <ul>
-   * <li>AUTO_FLUSH_SYNC: the call returns when the operation is persisted,
-   * else it throws an exception.
-   * <li>AUTO_FLUSH_BACKGROUND: the call returns when the operation has been added
to the buffer.
+   * <li>{@link SessionConfiguration.FlushMode#AUTO_FLUSH_SYNC AUTO_FLUSH_SYNC}:
+   * the call returns when the operation is persisted, else it throws an exception.
+   *
+   * <li>{@link SessionConfiguration.FlushMode#AUTO_FLUSH_BACKGROUND AUTO_FLUSH_BACKGROUND}:
+   * the call returns when the operation has been added to the buffer.
    * This call should normally perform only fast in-memory operations but
    * it may have to wait when the buffer is full and there's another buffer being flushed.
Row
    * errors can be checked by calling {@link #countPendingErrors()} and can be retrieved
by calling
    * {@link #getPendingErrors()}.
-   * <li>MANUAL_FLUSH: the call returns when the operation has been added to the buffer,
-   * else it throws a KuduException if the buffer is full.
+   *
+   * <li>{@link SessionConfiguration.FlushMode#MANUAL_FLUSH MANUAL_FLUSH}:
+   * the call returns when the operation has been added to the buffer, else it throws a
+   * {@link KuduException} if the buffer is full.
    * </ul>
    *
+   * <p>
+   * Note: {@link PleaseThrottleException} is handled by this method and will not be thrown,
unlike
+   * with {@link AsyncKuduSession#apply AsyncKuduSession.apply()}.
+   *
    * @param operation operation to apply
    * @return an OperationResponse for the applied Operation
    * @throws KuduException if anything went wrong
+   * @see SessionConfiguration.FlushMode FlushMode
    */
   public OperationResponse apply(Operation operation) throws KuduException {
     while (true) {

http://git-wip-us.apache.org/repos/asf/kudu/blob/97ad14d8/java/kudu-client/src/main/java/org/apache/kudu/client/SessionConfiguration.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/SessionConfiguration.java
b/java/kudu-client/src/main/java/org/apache/kudu/client/SessionConfiguration.java
index fe3485b..db6b398 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/SessionConfiguration.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/SessionConfiguration.java
@@ -31,33 +31,50 @@ public interface SessionConfiguration {
   @InterfaceAudience.Public
   @InterfaceStability.Evolving
   enum FlushMode {
-    // Every write will be sent to the server in-band with the Apply()
-    // call. No batching will occur. This is the default flush mode. In this
-    // mode, the Flush() call never has any effect, since each Apply() call
-    // has already flushed the buffer.
+    /**
+     * Each {@link KuduSession#apply KuduSession.apply()} call will return only after being
+     * flushed to the server automatically. No batching will occur.
+     *
+     * <p>In this mode, the {@link KuduSession#flush} call never has any effect, since
each
+     * {@link KuduSession#apply KuduSession.apply()} has already flushed the buffer before
+     * returning.
+     *
+     * <p><strong>This is the default flush mode.</strong>
+     */
     AUTO_FLUSH_SYNC,
 
-    // Apply() calls will return immediately, but the writes will be sent in
-    // the background, potentially batched together with other writes from
-    // the same session. If there is not sufficient buffer space, then Apply()
-    // may block for buffer space to be available.
-    //
-    // Because writes are applied in the background, any errors will be stored
-    // in a session-local buffer. Call CountPendingErrors() or GetPendingErrors()
-    // to retrieve them.
-    //
-    // The Flush() call can be used to block until the buffer is empty.
+    /**
+     * {@link KuduSession#apply KuduSession.apply()} calls will return immediately, but the
writes
+     * will be sent in the background, potentially batched together with other writes from
+     * the same session. If there is not sufficient buffer space, then
+     * {@link KuduSession#apply KuduSession.apply()} may block for buffer space to be available.
+     *
+     * <p>Because writes are applied in the background, any errors will be stored
+     * in a session-local buffer. Call {@link #countPendingErrors() countPendingErrors()}
or
+     * {@link #getPendingErrors() getPendingErrors()} to retrieve them.
+     *
+     * <p><strong>Note:</strong> The {@code AUTO_FLUSH_BACKGROUND} mode
may result in
+     * out-of-order writes to Kudu. This is because in this mode multiple write
+     * operations may be sent to the server in parallel.
+     * See <a href="https://issues.apache.org/jira/browse/KUDU-1767">KUDU-1767</a>
for more
+     * information.
+     *
+     * <p>The {@link KuduSession#flush()} call can be used to block until the buffer
is empty.
+     */
     AUTO_FLUSH_BACKGROUND,
 
-    // Apply() calls will return immediately, and the writes will not be
-    // sent until the user calls Flush(). If the buffer runs past the
-    // configured space limit, then Apply() will return an error.
+    /**
+     * {@link KuduSession#apply KuduSession.apply()} calls will return immediately, but the
writes
+     * will not be sent until the user calls {@link KuduSession#flush()}. If the buffer runs
past
+     * the configured space limit, then {@link KuduSession#apply KuduSession.apply()} will
return
+     * an error.
+     */
     MANUAL_FLUSH
   }
 
   /**
    * Get the current flush mode.
-   * @return flush mode, AUTO_FLUSH_SYNC by default
+   * @return flush mode, {@link FlushMode#AUTO_FLUSH_SYNC AUTO_FLUSH_SYNC} by default
    */
   FlushMode getFlushMode();
 
@@ -137,14 +154,15 @@ public interface SessionConfiguration {
    * This can be needed when facing KUDU-568. The effect of enabling this is that operation
    * responses that match this pattern will be cleared of their row errors, meaning that
we consider
    * them successful.
-   * This is disabled by default.
+   *
+   * <p>Disabled by default.
    * @param ignoreAllDuplicateRows true if this session should enforce this, else false
    */
   void setIgnoreAllDuplicateRows(boolean ignoreAllDuplicateRows);
 
   /**
    * Return the number of errors which are pending. Errors may accumulate when
-   * using the AUTO_FLUSH_BACKGROUND mode.
+   * using {@link FlushMode#AUTO_FLUSH_BACKGROUND AUTO_FLUSH_BACKGROUND} mode.
    * @return a count of errors
    */
   int countPendingErrors();
@@ -152,7 +170,8 @@ public interface SessionConfiguration {
   /**
    * Return any errors from previous calls. If there were more errors
    * than could be held in the session's error storage, the overflow state is set to true.
-   * Resets the pending errors.
+   *
+   * <p>Clears the pending errors.
    * @return an object that contains the errors and the overflow status
    */
   RowErrorsAndOverflowStatus getPendingErrors();

http://git-wip-us.apache.org/repos/asf/kudu/blob/97ad14d8/src/kudu/client/client.h
----------------------------------------------------------------------
diff --git a/src/kudu/client/client.h b/src/kudu/client/client.h
index 3c1f9b1..3255b0c 100644
--- a/src/kudu/client/client.h
+++ b/src/kudu/client/client.h
@@ -1222,6 +1222,14 @@ class KUDU_EXPORT KuduSession : public sp::enable_shared_from_this<KuduSession>
     /// the current batch is sent and the reclaimed space is available
     /// for new operations.
     ///
+    /// @attention The @c AUTO_FLUSH_BACKGROUND mode, when used in conjunction
+    ///   with a KuduSession::SetMutationBufferMaxNum() of greater than 1
+    ///   (the default is 2), may result in out-of-order writes. This
+    ///   is because the buffers may flush concurrently, so multiple write
+    ///   operations may be sent to the server in parallel.
+    ///   See [KUDU-1767](https://issues.apache.org/jira/browse/KUDU-1767) for
+    ///   more information.
+    ///
     /// @todo Provide an API for the user to specify a callback to do their own
     ///   error reporting.
     AUTO_FLUSH_BACKGROUND,
@@ -1229,6 +1237,15 @@ class KUDU_EXPORT KuduSession : public sp::enable_shared_from_this<KuduSession>
     /// Apply() calls will return immediately, and the writes will not be
     /// sent until the user calls Flush(). If the buffer runs past the
     /// configured space limit, then Apply() will return an error.
+    ///
+    /// @attention The @c MANUAL_FLUSH mode, when used in conjunction
+    ///   with a KuduSession::SetMutationBufferMaxNum() of greater than 1
+    ///   (the default is 2), may result in out-of-order writes if
+    ///   KuduSession::FlushAsync() is used. This is because the buffers may
+    ///   flush concurrently, so multiple write operations may be sent to the
+    ///   server in parallel.
+    ///   See [KUDU-1767](https://issues.apache.org/jira/browse/KUDU-1767) for
+    ///   more information.
     MANUAL_FLUSH
   };
 


Mime
View raw message