Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.219.205 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=DBaiemyOoer7NEZ5R86aQw/e4B5dHfi77ImuFjDUCy9uts5EL8QBf7Ql1pCzx8Wc4P
         GSAkzDgYNMre7jCzK2e3GUeDkt/juDGo7PJzj527WK5tED1wp17iOv8vkv/TAmz50kb5
         a3tFamq4jouqxtPlPhzj4tMv9tXAeekOe0mCA=
MIME-Version: 1.0
In-Reply-To: <f05b8c590910191820w234de060p7d92a534f9a2b10@mail.gmail.com>
References: <f05b8c590910191730n5581ff62te720e58a3fab8e6@mail.gmail.com>
	<e06563880910191736n2ecaa6f8u2ca0ba0e4b85898@mail.gmail.com>
	<f05b8c590910191740t5f731492l3e0111c6c1ef3923@mail.gmail.com>
	<e06563880910191801g3bfb7f66sc61ed61716d93c4c@mail.gmail.com>
	<f05b8c590910191820w234de060p7d92a534f9a2b10@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Mon, 19 Oct 2009 20:33:49 -0500
Message-ID: <e06563880910191833v316f4e2bq9ec871703bd08de5@mail.gmail.com>
Subject: Re: repeated timeouts on quorum reads
To: cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Oct 19, 2009 at 8:20 PM, Edmond Lau <edmond@ooyala.com> wrote:
> On Mon, Oct 19, 2009 at 6:01 PM, Jonathan Ellis <jbellis@gmail.com> wrote=
:
>> are there many rows like this?
>
> No - just a handful. =A0I'm able to repro by just launching 5 or 6
> threads that all read the same key.

Does it work if you just read one column? 2? 10?  If it does, how long
do those take?

Is the system busy with other reads or writes?

What resource do you see being used on the queried nodes?  cpu? (top)
disk? (iostat)

(Turn on debug logging to see which nodes are getting the queries, if
necessary.)

> I did find one point of note though. =A0Most of the quorum reads fail
> with 0 or 1 responses, but I see some that fail with 2 responses,
> which is odd given that only 2 responses were needed:

That means it got an answer from the 2 nodes computing hashes of the
answer for quorum purposes, but not the node that was tasked with
retrieving the actual data.

-Jonathan