db-derby-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Db-derby Wiki] Update of "Derby3192Writeup" by DyreTjeldvoll
Date Tue, 12 Feb 2008 14:48:48 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The following page has been changed by DyreTjeldvoll:

  == Implementing Piggy-backing ==
- /!\ '''The following paragraphs are mostly obsolete. The discussion at [https://issues.apache.org/jira/browse/DERBY-3192
+ /!\ '''The following paragraphs are still under construction. More details can be found
at [https://issues.apache.org/jira/browse/DERBY-3192 DERBY-3192].''' /!\
- has revealed that better implementation strategies probably exist. This page will be updated
- later.''' /!\
+ === Getting session data from the embedded driver === 
  In order for piggy-backing to be reasonably efficient the !NetworkServer needs an efficient
(and preferably) convenient way of obtaining the session data from its `EmbedConnection`,
preferably by providing a `getCurrentSchemaName()` method. It would of course be possible
to implement this by executing `VALUES CURRENT ISOLATION` on the embedded connection, but
this seems like much overhead, especially when considering that this query will have to be
performed whenever the !NetworkServer needs to know if the session data has changed.
  An even more efficient approach would be if the NetworkServer could register itself as some
kind of listener with `EmbedConnection` which, in turn, would invoke some kind of callback
which informed the !NetworkServer that session data had been modified. This advantage of this
approach did not seem to justify the added complexity and changes to the embedded driver.
+ === Identifying session data changes ===
  Given that we have chosen to poll for changes to the session data in the `EmbedConnection`
each time they may have changed, we also need the ability to store the last
  values sent to the client, and a way to compare those with the current values from `EmbedConnection`.

- == Piggy-backing in DRDA ==
+ === Encoding and decoding session data ===
+ === Maintaining the cached session data on the client side ===
- DRDA provides the EXCSQLSET command to "batch" (for lack of a better
- term) SET statements from a client to the server. This command is
- currently used to implement the `setQueryTimeout()` functionality. This DRDA command can
also be used to poll for a changed
- isolation level. The client code provides a method
- (`Statement.writeSetSpecialRegister()`) for adding an EXCSQLSET command
- to a request. This is really a family of methods in various classes
- and interfaces with the most action concentrated in
- `Statement.wrteSetSpecialRegister` and
- `NetStatementRequest.writeSetSpecialRegister()`.
+ == Handling XA Transactions ==
- `setQueryTimeout()` implements its functionality by letting this command
- send a special statement which also encodes the timeout value to
- set. Session data polling uses the same idea but with a different dummy
- statement which, just like in the `setQueryTimeout()` case, is intercepted
- by the server and given special treatment.
+ == Overhead ==
- A new method is added to `EmbedConnection` and also to the
- `EngineConnection` interface (and thereby also to the `BrokeredConnection`
- interface), that lets the caller iterate over the connection
- attributes that have changed since last time this method was called,
- (Only the isolation level will be reported initially, but this should be extended to other
parameters, such as the current schema,
- later).
+ Piggy-backing will necessarily add some overhead to the replies being sent to the client,
but there is no overhead on the requests from the client to server (unlike in the previous
patch proposal).
- When the server receives the polling request it always adds
- the SQLCARD required by EXCSQLSET to the reply. If a cacheable connection
- attribute has changed it is added to the reply in the form of
- an SQLSTT codepoint containing a string with the following format:
- 'key=value', where 'key' identifies the attribute that has changed,
- and 'value' is its new value.
- There is a `readSetSpecialRegister()` method (corresponding to
- `writeSetSpecialRegister()`), which will read the reply on the client
- side, and invoke a callback (`completePiggyBackSessionData()`) on the
- `am.Connection` object through
- `ConnectionCallbackInterface.completePiggyBackSessionData()` will
- update the relevant attributes (currently only
- `am.Connection.isolation_`).
- === Use of the RTNSETSTT instance variable ===
- The DRDA spec specifies that EXCSQLSET can have an optional
- instance variable RTNSETSTT, that essentially is a boolean flag which
- allows the requester (the client) to tell the server that it expects
- (and can handle) SQLSTT code points in the reply. Since 
- such SQLSTT code points are used to transmit the cached attributes "diff", it
- is also necessary to set the RTNSETSTT instance variable when sending the
- request. Note that this is essentially "redundant" information since
- the server knows exactly what to do as soon as it has identified the
- special "statement" used in the polling request. It is only added to
- conform to the DRDA spec. 
- One could also have omitted the use of RTNSETSTT and rather encoded
- the connection attributes in the mandatory SQLCARD. The SQLCARD is
- required but will never be used, since the polling request will not
- actually execute a SET statement that can fail. 
- == Overhead Introduced by Piggy-backing ==
- As mentioned previously, the polling approach will necessarily
- introduce some overhead both on requests and replies. 
- === Request ===
- One EXCSQLSET command is added to every message containing OPNQRY and
- CNTQRY commands. Most of the overhead seems to come
- from the PKGNAMCSN instance variable which by itself uses 76 bytes, or
- 59% of the total overhead:
- || '''DRDA VARIABLE''' || '''SIZE''' (BYTES) ||
- || DSS header || 6  ||
- || EXCSQLSET  || 4  ||
- || PKGNAMCSN  || 76 ||
- || RTNSETSTT  || 4  ||
- || HEXSTRDR   || 5  ||
- || SQLSTT     || 4 + 1 + 4 + 23(strlen) + 1 = 33 ||
- || TOTAL      || 128 ||
- This gives a total overhead of (a whopping) 128 bytes. 
- It can be reduced by shortening the name used for the "special
- statement", but the overhead would still be at least 105 bytes. 
- If we also omit the RTNSETSTT instance variable the overhead could be 
- further reduced to 96 bytes per request. Which is still substantial.
  === Reply ===
+ Separate DSS(6) + two cp (8) + pay (1->)
- The only mandatory variable in an EXCSQLSET reply is SQLCARD. When
- used to poll for modified session data, this variable has no real
- meaning, and null is passed in as the SQLException parameter to
- `DDMWriter.writeSQLCARD()`. Initially is was (incorrectly) assumed
- that passing null would result in a minimal increase of the
- message size. In reality, the DSS size (as reported by
- `getDSSLength()`) increases to 65 after adding the SQLCARD (which creates
- its own DSS):
- || '''DRDA VARIABLE''' || '''SIZE''' (BYTES) ||
+ || '''DRDA VARIABLE''' || '''SIZE''' (BYTES)           ||
+ || DSS (PBSD)          || 6                                  ||
+ ||  PBSD_ISO           || 4 + 1                              ||
+ ||  PBSD_SCHEMA        || 4 + `bytelen(UTF8(schemaname))`     ||
+ || TOTAL               || 11/15 + `bytelen(UTF8(schemaname))` ||
- || DSS header    || 6            ||
- ||  SQLCARD      || 65           ||
- ||  SQLSTT       || 4 + 1 + 4 + 5(strlen) + 1 = 15 ||
- ||   ...         ||  ...         ||
- || TOTAL         || 71/86        ||
+ When no session parameter has changed (the common case) no overhead is introduced. If only
the isolation level has changed 11 additional bytes are added to the next reply. If only the
schema has been modified 10 bytes + the byte length of the UTF-8 representation of the schema
name are added. In the worst case, when both attributes have changed, 15 bytes + the byte
length of the schema name will be added.
- When no session parameter has changed (the common case) only the 71
- bytes of the DSS header and the SQLCARD will be added to the reply. If
- one or more session parameters have changed, one SQLSTT object will be
- added for each changed parameter. The size of the SQLSTT obviously depends on the
- value of the parameter, but since SQLSTTs are strings the byte count
- will also depend on whether single-byte or multi-byte encoding is used
- for the string representing the parameter value. 
- = Multiple Round-trip Replies =
+ == Compatibility ==
- Following the example of `setQueryTimeout()`, calls to
- `writeSetSpecialRegister()` and `readSetSpecialRegister()` are made
- inside `Statement.flowExecute()`. The request is added in the first
- part of the method, prior to calling `agent_.flow()`, and the reply is
- read in the latter part of the method, after the blocking call to
- `agent_.flow()` returns.
- In the case of `setQueryTimeout()`, the request is added to the
- beginning of the outgoing message (which is reasonable, given that it needs to ensure that
- all the following actions observe the time limit), and the subsequent
- reply is the first received and parsed after `agent_.flow()` returns. 
- When polling for changed session data the situation is the exact
- opposite; the poll request needs to be the last thing added to the
- outgoing message to ensure that the poll picks up any changes in the
- session state that has happened as a consequence of the other commands
- in this message. As a result, the reply must be received and parsed at
- the end of `agent_.flow()`, after all the other replies have been parsed.
- == Scrollable Result Sets ==
- Most of the time this works well because call to `agent_.flow()` is
- followed by a single reply, but for scrollable
- result sets this is not the case. If `Statement.isQuery__` is
- true and the `Statement.resultSet_` is not null, a call to
- `parseScrollableRowset()` is made:
- {{{#!java
- if (resultSet_ != null) {
-     resultSet_.parseScrollableRowset();
- }}}
- This happens irrespective of the type of the result set, but the call
- to `parseRowset_()` inside `am.ResultSet.parseScrollableRowset()` is
- only made if `scrollable_` is true:
- {{{#!java
- if (cursor_.dataBufferHasUnprocessedData() && scrollable_) {
-     parseRowset_();
- }}}
- `net.NetResultSet.parseRowSet_()` ''may'' call
- `flowFetchToCompleteRowset()` directly, or through
- `calculateColumnOffsetsForRow_()`, and this call initiates another
- roundtrip, while still inside `Statement.flowExecute()`:
- {{{#!java
- void flowFetchToCompleteRowset() throws DisconnectException {
- try {
-     agent_.beginWriteChain(statement_);
-     writeScrollableFetch_((generatedSection_ == null) ? statement_.section_ : generatedSection_,
-         fetchSize_ - rowsReceivedInCurrentRowset_,
-         scrollOrientation_relative__,
-         1,
-         false);  // false means do not disard pending
-     // partial row and pending query blocks
-     agent_.flow(statement_);
-     readScrollableFetch_();
-     agent_.endReadChain();
- }}}
- This is problematic because:
-  * The QRYDTA parsing does not expect anything to follow the last
-  QRYDTA reply in a message, so if all QRYDTAs have not been received,
-  it will fail when reading the poll reply.
-  * The new round-trip (CNTQRY) initiated by the parsing code will not
-  add a new session data poll request, so even if the poll reply
-  is handled or ignored by the QRYDTA parsing, there will be no
-  poll reply to read when QRYDATA parsing is complete and flow control
-  reaches `readSetSpecialRegister()` in 
-  `Statement.flowExecute()`, and the call will block forever.
- The problem is limited to '''read only''' result sets since '''updatable'''
- result sets always have a fetch size of 1, regardless of what the
- suggested fetch size is.
- == Possible Solutions ==
- The initial idea was to handle this by trying to detect this situation
- on the server, and not send a poll reply until all calls to
- `flowFetchToCompleteRowset()` had been completed, so that the
- poll reply would be handled correctly when control returned to
- `Statement.flowExecute()`.
- This approach does not work correctly when `fetchsize_` is smaller
- than the number of CNTQRYs needed to get a complete row set, because
- then the loop calling `flowFetchToCompleteRowset()` will terminate
- before the server sends the poll reply, and then the code
- inside `Statement.flowExecute()` trying to read the reply will
- hang waiting for data which will never arrive.
- At this point the best solutions seem to be: 
-  1. Add a poll request command to every CNTQRY and parse it in every reply (also those handled
by `flowFetchToCompleteRowSet()`).
-  1. Implement some kind of bookkeeping mechanism which tracks if a poll request was included
in the last message sent, and if any poll reply in the last incoming message has been handled.
This adds complexity, but reduces the number of unnecessary poll request/replies in some situations.

View raw message