zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miller, Austin" <Austin.Mil...@morganstanley.com>
Subject RE: exposing lastSend
Date Mon, 21 Jul 2014 15:07:23 GMT

Some links to give context,

(1) https://github.com/apache/zookeeper/blob/b79af153d0f98a4f3f3516910ed47234d7b3d74e/src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java#L178-188
(2) https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/ClientCnxnSocketNIO.java#L99-102
(3) https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/ClientCnxn.java#L826-836
(4) https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/ClientCnxnSocketNIO.java#L111-112

Based on link #4, I think now that after a long GC pause, the SendThread would (or at least
could) race to update the last send with any thread trying to gate writes using the value.
 This update of last send occurs before the socket IO, so wouldn't fail to happen even if
the OS thought the socket was in an exceptional state.  So, in light of this, my earlier idea
is not going to work.

But, I noticed that the server is returning an application level heartbeat based on link #1.
 And, based on link #2, this causes a "lastHeard" state to be registered.  Also, based on
link #3, the "latency" of the ping is logged, but not recorded.

So, I've come up with a new proposal:

  * Create a ClientCnxnStat object.
  * Move all the current recorded state (packet stats, lastHeard, lastSend, etc) to ClientCnxnStat.
  * Also start recording the lastPingResponse in the location of link #3.
  * All the values could be volatile, the object only possessing getters.
  * Package private in org.apache.zookeeper to allow the connection objects to update the
  * alternatively, the method of exposure could deep copy the object, but some synchronization
would be needed to get a "snapshot" effect.
  * Expose the ClientCnxnStat object via zookeeper using ZooKeeper.getClientCnxnStat().

This opens up many possibilities, like exposing some programmatic insight into the health
of the network connection with the ensemble.  For example, it would be possible to calculate
the latency of pings.

And, I, and others concerned about transactions in the presence of long GC pauses, could use
a combination of lastHeard and lastPingResponse to gate transaction writes and increase confidence.

I anticipate some concerns, like it might degrade performance by slowing down by needing to
sync via volatile or a ReadWriteLock to get a snapshot effect.




NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies and notify the
sender immediately. Mistransmission is not intended to waive confidentiality or privilege.
Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor
electronic communications. This message is subject to terms available at the following link:
http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify
us by reply message and we will send the contents to you. By messaging with Morgan Stanley
you consent to the foregoing.
View raw message