cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Very slow reads - connected to TBufferedTransport buffer sizes
Date Tue, 03 Aug 2010 12:18:22 GMT
Sounds like https://issues.apache.org/jira/browse/THRIFT-638, where
Arya Goudarzi posted a patch.

On Tue, Aug 3, 2010 at 5:18 AM, Dave Gardner <dave.gardner@imagini.net> wrote:
> Hi all
>
> I'm working on a PHP/Cassandra application. Yesterday we experienced a
> strange situation when testing random reads. The background to this test was
> that we inserted 10,000 rows with simple row keys. The number of columns in
> each row varies between about 5 columns and 40 columns (all random).  Insert
> speed via the PHP Thrift library was good - very fast.
>
> With our random read script, we are simply carrying out a get_slice on a
> specific row key to fetch all the columns in the row (up to 100, which is
> always everything).
>
> The random read script either completes in a few ms, or it takes around 5s.
> The significance of 5s is that this is (temporarily) what we've set the
> socket read timeout to (TSocket::setRecvTimeout). There are a couple of
> interesting things to note:
>
> 1. Even when the socket seemingly times out, the result of the operation is
> successful - eg: we do end up with a row returned.
> 2. The slow response times are *always* for rows with a larger number of
> columns; the limit seems to be around 9 columns or so
> 3. If we up the timeout to 10s, the reads take 10s instead of 5s, but still
> return successfully
>
> This is an example of data from a row that is read quickly
>
> 9bc905c5-fc62-58de-87f5-48eb1ebb4f03
>     col0: e0ce34010211030d91331feefc946c8c
>     col1: 6d97b892ee7773d40b7bbff27ec5b34d
>     col2: 6e2394dd48d5ca2df47eeceb72ca9de0
>     col3: 43fca4716b865f24e30de67b2e10c1a8
>     col4: c2e8de1541550e78829d312609acd237
>     col5: e458447d8a2987bf05d65bee0a103be8
>     col6: 0a8a86de4247b690e765aeca6615aef8
>     col7: dc48b5e996da86b94d40d85292351c61
>     col8: 3b95f9fc7c64d021ecc2c7a013f2e132
>     key: 9bc905c5-fc62-58de-87f5-48eb1ebb4f03
>
> This is an example of data from a row that is read slowly:
>
> 7318f337-529d-5408-a7c8-1283b750164d
>     col0: 113b9cfe8eea8bf7eca71ce1ca1b0913
>     col1: 428fe0bfadf687ef3b5c532e98e487ef
>     col10: f1e80507626223358414130b1c7ecacd
>     col11: cf7ada7ab098d2aeb9e5553808c89044
>     col12: a93237313167c313d36d39779dcf23cd
>     col13: 609595bbbb2b7058ad3f97f1ea0b7ebd
>     col2: 27eca7dbff849eac82dc32e92b3fe977
>     col3: 294dbf3107c351783a69450fedbefc61
>     col4: 7fbd8f20d52731a10029e6f92874fae5
>     col5: 3d06fc491c8f1669b144b798155578d4
>     col6: 60d8f358cf07924912c8e19c60f45aac
>     col7: 03297fd9576c1c96586bbbaaeaa1aa64
>     col8: 6d6383fba84a6ec6811d96aee2b39102
>     col9: c937171c3ad5d30b671d72741a777dc7
>     key: 7318f337-529d-5408-a7c8-1283b750164d
>
> Other points:
>
> - we are reading with CL:One
> - we are using the PHP Thrift library directly
> - we are using:
>    TSocket
>    TBufferedTransport                  [with buffer sizes of 1024,
1024]
>    TBinaryProtocolAccelerated
>
> I have just this second discovered that changing the buffer sizes impacts
> this issue. Reducing the buffer size makes every request take 5s, increasing
> the buffer size makes every request execute quickly.
>
> I will continue to debug this issue today, but thought that someone may be
> able to shed some light on the issue.
>
> Thanks!
>
> Dave
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message