zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miller, Austin" <Austin.Mil...@morganstanley.com>
Subject exposing lastSend
Date Fri, 11 Jul 2014 17:31:15 GMT
Hello,

I'm looking for a way to get access to ClientCnxnSocket.lastSend without trying to break ZooKeeperEncapsulation.
 Broadly, my goal is to use this in a Java process in order to increase confidence in transactions.

There are resource starved situations where the ClientCnxn.SendThread may not be scheduled
for greater than negotiatedSessionTimeout.  My understanding is this will lead to session
loss because even though the connection might still be considered alive (the OS could be sending
ACKs to packets ZK sends), the ZK server requires the client to be sending the packets.

So, assuming
...a JVM process was connected to a ZK ensemble
...the JVM process is performing transactions
...ZK is being used for distributed locking with coarse granularity
...a reliable low-latency network connection to a healthy low-latency ensemble
...a rare event causes the machine hosting the JVM to be resource starved
...none of the JVM threads are scheduled for a window twice the length of the negotiatedSessionTimeout
...during this window, the process has lost the coarse lock on the ensemble (it was an ephemeral
node)

Then the ensemble should have agreed that the session is dead, correct?  Even though the connection
may be considered alive at a TCP/IP transport level.  What is more, just coming out of the
state where the threads are scheduled, there is a race condition between the ZK threads firing
session death event and the transaction threads committing transactions.  As I write this,
I realize I'm not entirely sure what events ZK would send and in what order, it depending
on what was done before the freeze and where it was frozen.

Back to the broad goal, I want to increase confidence in this situation that the process still
owns the ZK lock without firing off network events before committing every transaction.  Obviously,
fine granular locks would solve this problem, but that comes with an unacceptable performance
trade off.

Now, let's say I could do something like "long org.apache.zookeeper.ZooKeeper.getLastSent()".
 Well, I don't know if the ZK server actually received the packet, assuming it did receive
the packet I don't know when it received the packet, and I don't know when the OS received
the ack.  However, it does assert that the SendThread was scheduled and able to call System.getNanos()
in ClientCnxnSocket.  This increasing the likelihood that the process was sending heartbeats.
 In addition to this, if I haven't received a push notification from the ZK event thread implying
I've lost the lock, I have higher confidence that the session hasn't been lost and that I
still have the coarse lock, which satisfies my broad goal somewhat better than the current
state.

Any thoughts?

Austin


________________________________

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies and notify the
sender immediately. Mistransmission is not intended to waive confidentiality or privilege.
Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor
electronic communications. This message is subject to terms available at the following link:
http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us
by reply message and we will send the contents to you. By messaging with Morgan Stanley you
consent to the foregoing.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message