I'm working on a PHP/Cassandra application. Yesterday we experienced a strange situation when testing random reads. The background to this test was that we inserted 10,000 rows with simple row keys. The number of columns in each row varies between about 5 columns and 40 columns (all random). Insert speed via the PHP Thrift library was good - very fast.
With our random read script, we are simply carrying out a get_slice on a specific row key to fetch all the columns in the row (up to 100, which is always everything).
The random read script either completes in a few ms, or it takes around 5s. The significance of 5s is that this is (temporarily) what we've set the socket read timeout to (TSocket::setRecvTimeout). There are a couple of interesting things to note:
1. Even when the socket seemingly times out, the result of the operation is successful - eg: we do end up with a row returned.
2. The slow response times are *always* for rows with a larger number of columns; the limit seems to be around 9 columns or so
3. If we up the timeout to 10s, the reads take 10s instead of 5s, but still return successfully
This is an example of data from a row that is read quickly
This is an example of data from a row that is read slowly:
- we are reading with CL:One
- we are using the PHP Thrift library directly
- we are using:
TBufferedTransport [with buffer sizes of 1024, 1024]
I have just this second discovered that changing the buffer sizes impacts this issue. Reducing the buffer size makes every request take 5s, increasing the buffer size makes every request execute quickly.
I will continue to debug this issue today, but thought that someone may be able to shed some light on the issue.