incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: repeated timeouts on quorum reads
Date Tue, 20 Oct 2009 03:04:45 GMT
On Mon, Oct 19, 2009 at 9:48 PM, Edmond Lau <edmond@ooyala.com> wrote:
> 8 data files total.  3 nodes have 1, 1 has 2, the 3rd has 3.
>
>> Does it still take ~8s if you direct a CL.ONE query at one of the
>> nodes you know has the data (i.e., a local read)?
>
> Local reads return quickly, but if you look at the logs below, there's
> 6 secs from when cassandra prints that it's reading a consistency
> digest to when it processes a callback.

That's from when it sends the digest command, to when it gets a callback.

Something doesn't add up.  If computing &sending a digest response
takes 6s, either reading the row or computing the digest must be
taking substantial time.  But if local reads are fast, it's not
reading the row that is slow, and if resolve is fast (from your logs,
it is 2ms) then it's not computing the digest that is slow.

Try doing a local read from the node with 3 data files, see if that is slower.

If you still see a fast read there, add some more debug logging in
ReadVerbHandler.java, where it says something like (I'm going from
trunk here but 0.4.1 is similar)

            Row row = readCommand.getRow(table);
            ReadResponse readResponse;
            if (readCommand.isDigestQuery())
            {
                readResponse = new ReadResponse(row.digest());

time the getRow and the row.digest() calls and log how long those take
in the quorum scenario.

-Jonathan

Mime
View raw message