cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jun Rao (JIRA)" <>
Subject [jira] Updated: (CASSANDRA-220) TCP writes get stuck
Date Tue, 23 Jun 2009 16:30:07 GMT


Jun Rao updated CASSANDRA-220:

    Attachment: 220-4.patch

It took me a while to dig this out. Under heavy load and large column values, we still saw
lockups in tcp connection. Here is the problem. The following code that sets the interest
ops seems innocent, but it's the source of the problem. The reason is that this operation
is not atomic. Another thread could sneak in between the reading of the ops and the setting
of it. As a result, some wrong bits could be set.
   key_.interestOps(key_.interestOps() | SelectionKey.OP_READ)

This is a sequence that demonstrates how we can lose the OP_READ bit forever and thus jam
the read channel:
1. Thread 1: we want to write a message and in write(Message) we are about to turn on OP_WRITE
because the message can't be written in one shot.
2. Thread 2: a read comes in and in read(SelectionKey), we turn off OP_READ and submit the
read request to ReadWorkItem in Thread 3.
3. Thread 1: read interestOps and see OP_READ as off.
4. Thread 3: finished processing the read request and turn OP_READ on
5. Thread 1: resumes and turn on OP_WRITE. However, by doing that, we also turned off OP_READ.
The read channel is thus blocked forever after this.

Patch-4 makes changing interestOps atomic by synchronizing on the selection key.

Patch-4 also includes the earlier fix for the synchronization problem in connect. I left the
fix as it is instead of making connect() a synchronized method. This way, it makes clear which
part of the code in connect really requires synchronization.

> TCP writes get stuck
> --------------------
>                 Key: CASSANDRA-220
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jun Rao
>            Assignee: Jun Rao
>         Attachments: 220-2.patch, 220-3.patch, 220-4.patch, issue220.patchv1
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites.
However, the selector never gets a ready to write event. As a result, all writes get stuck.
This is because write(message) and doPendingWrites() are not fully synchronized.  This following
situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written
to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty()
is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message