cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Gardner <dave.gard...@imagini.net>
Subject Re: Very slow reads - connected to TBufferedTransport buffer sizes
Date Tue, 03 Aug 2010 18:50:50 GMT
Yes this is the issue. Thanks.

Dave

On Tuesday, August 3, 2010, Jonathan Ellis <jbellis@gmail.com> wrote:
> Sounds like https://issues.apache.org/jira/browse/THRIFT-638, where
> Arya Goudarzi posted a patch.
>
> On Tue, Aug 3, 2010 at 5:18 AM, Dave Gardner <dave.gardner@imagini.net> wrote:
>> Hi all
>>
>> I'm working on a PHP/Cassandra application. Yesterday we experienced a
>> strange situation when testing random reads. The background to this test was
>> that we inserted 10,000 rows with simple row keys. The number of columns in
>> each row varies between about 5 columns and 40 columns (all random).  Insert
>> speed via the PHP Thrift library was good - very fast.
>>
>> With our random read script, we are simply carrying out a get_slice on a
>> specific row key to fetch all the columns in the row (up to 100, which is
>> always everything).
>>
>> The random read script either completes in a few ms, or it takes around 5s.
>> The significance of 5s is that this is (temporarily) what we've set the
>> socket read timeout to (TSocket::setRecvTimeout). There are a couple of
>> interesting things to note:
>>
>> 1. Even when the socket seemingly times out, the result of the operation is
>> successful - eg: we do end up with a row returned.
>> 2. The slow response times are *always* for rows with a larger number of
>> columns; the limit seems to be around 9 columns or so
>> 3. If we up the timeout to 10s, the reads take 10s instead of 5s, but still
>> return successfully
>>
>> This is an example of data from a row that is read quickly
>>
>> 9bc905c5-fc62-58de-87f5-48eb1ebb4f03
>>     col0: e0ce34010211030d91331feefc946c8c
>>     col1: 6d97b892ee7773d40b7bbff27ec5b34d
>>     col2: 6e2394dd48d5ca2df47eeceb72ca9de0
>>     col3: 43fca4716b865f24e30de67b2e10c1a8
>>     col4: c2e8de1541550e78829d312609acd237
>>     col5: e458447d8a2987bf05d65bee0a103be8
>>     col6: 0a8a86de4247b690e765aeca6615aef8
>>     col7: dc48b5e996da86b94d40d85292351c61
>>     col8: 3b95f9fc7c64d021ecc2c7a013f2e132
>>     key: 9bc905c5-fc62-58de-87f5-48eb1ebb4f03
>>
>> This is an example of data from a row that is read slowly:
>>
>> 7318f337-529d-5408-a7c8-1283b750164d
>>     col0: 113b9cfe8eea8bf7eca71ce1ca1b0913
>>     col1: 428fe0bfadf687ef3b5c532e98e487ef
>>     col10: f1e80507626223358414130b1c7ecacd
>>     col11: cf7ada7ab098d2aeb9e5553808c89044
>>     col12: a93237313167c313d36d39779dcf23cd
>>     col13: 609595bbbb2b7058ad3f97f1ea0b7ebd
>>     col2: 27eca7dbff849eac82dc32e92b3fe977
>>     col3: 294dbf3107c351783a69450fedbefc61
>>     col4: 7fbd8f20d52731a10029e6f92874fae5
>>     col5: 3d06fc491c8f1669b144b798155578d4
>>     col6: 60d8f358cf07924912c8e19c60f45aac
>>     col7: 03297fd9576c1c96586bbbaaeaa1aa64
>>     col8: 6d6383fba84a6ec6811d96aee2b39102
>>     col9: c937171c3ad5d30b671d72741a777dc7
>>     key: 7318f337-529d-5408-a7c8-1283b750164d
>>
>> Other points:
>>
>> - we are reading with CL:One
>> - we are using the PHP Thrift library directly
>> - we are using:
>>    TSocket
>>    TBufferedTransport                  [with buffer sizes of 1024,
1024]
>>    TBinaryProtocolAccelerated
>>
>> I have just this second discovered that changing the buffer sizes impacts
>> this issue. Reducing the buffer size makes every request take 5s, increasing
>> the buffer size makes every request execute quickly.
>>
>> I will continue to debug this issue today, but thought that someone may be
>> able to shed some light on the issue.
>>
>> Thanks!
>>
>> Dave
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message