Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 398D51748C for ; Wed, 11 Mar 2015 01:00:09 +0000 (UTC) Received: (qmail 70367 invoked by uid 500); 11 Mar 2015 01:00:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 70325 invoked by uid 500); 11 Mar 2015 01:00:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 70315 invoked by uid 99); 11 Mar 2015 01:00:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Mar 2015 01:00:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [209.85.160.169] (HELO mail-yk0-f169.google.com) (209.85.160.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Mar 2015 00:59:55 +0000 Received: by ykr79 with SMTP id 79so2469005ykr.13 for ; Tue, 10 Mar 2015 17:57:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:sender:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=Dgl4fMw5KkGGl/gOuosPO+q3tkE7bjDX6gHnnOQ3pgE=; b=IoxDRWvMLtL6Czc64Nk5zwVoa4rwfDifnQflOa5v+Kx0+lv5Hy6iGfZeL0GmdK4UFX L3RM5d28WjcXzp++P5FbwG67526gkC64g7QcPrfcd9lK9OLessq/oZspOdnkmz9tnpTc aFXN2CHLzpNu0tDmuD0rM3GPhSrc3X5+xuphbVSYIf0Ptxk6YV6ons5MczCH20n1msGn W8zlj3nVJh09WToNOp0fxy/1NySFHRaDOwwcvwR0cJ/XyFdCzQdkVnMCFaSJgpv27Usy Q4BqNZtkj3czBwlCjkWXXF2ccR5FO6rb51leGpm+96PINX10+y5s4XrItl0xsNI5on/p wCCw== X-Gm-Message-State: ALoCoQlHWUK75+IenmzdEdj+rl25lYDQBI3/9EPs4FOvLk4HP0P+hA4c+nIUQ2JTK40AWHWOZkUY MIME-Version: 1.0 X-Received: by 10.236.207.134 with SMTP id n6mr33890720yho.42.1426035464038; Tue, 10 Mar 2015 17:57:44 -0700 (PDT) Reply-To: mail@frensjan.nl Sender: frensjan@frensjan.nl Received: by 10.170.83.195 with HTTP; Tue, 10 Mar 2015 17:57:43 -0700 (PDT) X-Originating-IP: [94.212.67.32] In-Reply-To: References: <1425492417697.a19cfa28@Nodemailer> <1425495806878.c9686778@Nodemailer> Date: Wed, 11 Mar 2015 01:57:43 +0100 X-Google-Sender-Auth: J_yg-tPB8SjKwocnQZrug2vD7PM Message-ID: Subject: Re: Inconsistent count(*) and distinct results from Cassandra From: "Rumph, Frens Jan" To: DuyHai Doan Cc: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e016338064585ad0510f8c1e7 X-Virus-Checked: Checked by ClamAV on apache.org --089e016338064585ad0510f8c1e7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks for the suggestion DuyHai. I assume you mean CL=3DQUORUM (as in consistency level, not replication factor). As expected, setting the consistency level to quorum or all yields equally inconsistent results for the select count and select distinct queries. Which is good in a way, because if RF=3D1 and CL=3DONE I would expect an er= ror if one of the nodes wouldn't be able to answer a query. Note that there conceptually is no such thing as a quorum or majority when RF=3D1. As quorum in C* is defined as floor( (RF / 2) + 1 ), in case of RF= =3D1 this in practice is the same as CL=3DONE. On 10 March 2015 at 18:10, DuyHai Doan wrote: > First idea to eliminate any issue with regards to staled data: issue the > same count query with RF=3DQUORUM and check whether there are still > inconsistencies > > On Tue, Mar 10, 2015 at 9:13 AM, Rumph, Frens Jan > wrote: > >> Hi Jens, Mikhail, Daemeon, >> >> Thanks for your replies. Sorry for my reply being late ... mails from th= e >> user-list were moved to the wrong inbox on my side. >> >> I'm in a development environment and thus using replication factor =3D 1 >> and consistency =3D ONE with three nodes. So the 'results from different >> nodes between queries' hypothesis seems unlikely to me. I would expect a >> timeout if some node wouldn't be able to answer. >> >> I tried tracing, but I couldn't really make any of it. >> >> For example I performed two select distinct ... from ... queries: Traces >> for both of them contained more than one line like 'Submitting range >> requests on ... ranges ...' and 'Submitted ... concurrent range requests >> covering ... ranges'. These lines occur with varying numbers, e.g. : >> >> Submitting range requests on 593 ranges with a concurrency of 75 (1.35 >> rows per range expected) >> Submitting range requests on 769 ranges with a concurrency of 75 (1.35 >> rows per range expected) >> >> >> Also when looking at the lines like 'Executing seq scan across ... >> sstables for ...' I saw that in one case which yielded way less partitio= n >> keys that only the tokens from -9223372036854770000 to -594461978511041= 000 >> were included. In a case which yielded much more partition keys, the ent= ire >> token range did seem to be queried. >> >> To reiterate my initial questions: is this behavior to be expected? Am I >> doing something wrong? Is there a workaround? >> >> Best regards, >> Frens Jan >> >> On 4 March 2015 at 22:59, daemeon reiydelle wrote: >> >>> What is the replication? Could you be serving stale data from a node >>> that was not properly replicated (hints timeout exceeded by a node bein= g >>> down?) >>> >>> >>> >>> On Wed, Mar 4, 2015 at 11:03 AM, Jens Rantil >>> wrote: >>> >>>> Frens, >>>> >>>> What consistency are you querying with? Could be you are simply >>>> receiving result from different nodes each time. >>>> >>>> Jens >>>> >>>> =E2=80=93 >>>> Skickat fr=C3=A5n Mailbox >>>> >>>> >>>> On Wed, Mar 4, 2015 at 7:08 PM, Mikhail Strebkov >>>> wrote: >>>> >>>>> We have observed the same issue in our production Cassandra cluster (= 5 >>>>> nodes in one DC). We use Cassandra 2.1.3 (I joined the list too late = to >>>>> realize we shouldn=E2=80=99t user 2.1.x yet) on Amazon machines (crea= ted from >>>>> community AMI). >>>>> >>>>> In addition to count variations with 5 to 10% we observe variations >>>>> for the query =E2=80=9Cselect * from table1 where time > '$fromDate' = and time < >>>>> '$toDate' allow filtering=E2=80=9D results. We iterated through the r= esults >>>>> multiple times using official Java driver. We used that query for a h= uge >>>>> data migration and were unpleasantly surprised that it is unreliable.= In >>>>> our case =E2=80=9Cnodetool repair=E2=80=9D didn=E2=80=99t fix the iss= ue. >>>>> >>>>> So I echo Frens questions. >>>>> >>>>> Thanks, >>>>> Mikhail >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 4, 2015 at 3:55 AM, Rumph, Frens Jan >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Is it to be expected that select count(*) from ... and select >>>>>> distinct partition-key-columns from ... to yield inconsistent result= s >>>>>> between executions even though the table at hand isn't written to? >>>>>> >>>>>> I have a table in a keyspace with replication_factor =3D 1 which is >>>>>> something like: >>>>>> >>>>>> CREATE TABLE tbl ( >>>>>> id frozen, >>>>>> bucket bigint, >>>>>> offset int, >>>>>> value double, >>>>>> PRIMARY KEY ((id, bucket), offset) >>>>>> ) >>>>>> >>>>>> The frozen udt is: >>>>>> >>>>>> CREATE TYPE id_type ( >>>>>> tags map >>>>>> ); >>>>>> >>>>>> When I do select count(*) from tbl several times the actual count >>>>>> varies with 5 to 10%. Also when performing select distinct id, bucke= t from >>>>>> tbl the results aren't consistent over several query executions. The= table >>>>>> is not being written to at the time I performed the queries. >>>>>> >>>>>> Is this to be expected? Or is this a bug? Is there a alternative >>>>>> method / workaround? >>>>>> >>>>>> I'm using cqlsh 5.0.1 with Cassandra 2.1.2 on 64bit fedora 21 with >>>>>> Oracle Java 1.8.0_31. >>>>>> >>>>>> Thanks in advance, >>>>>> Frens Jan >>>>>> >>>>> >>>>> >>>> >>> >> > --089e016338064585ad0510f8c1e7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for the suggestion DuyHai. I assume you mean CL=3DQ= UORUM (as in consistency level, not replication factor). As expected, setti= ng the consistency level to quorum or all=C2=A0yields equally inconsistent = results=C2=A0for the select count and select distinct queries.

Which is good in a way, because if RF=3D1 and CL=3DONE I would expect an e= rror if one of the nodes wouldn't be able to answer a query.=C2=A0

Note that there conceptually is no such thing as a quo= rum or majority when RF=3D1. As quorum in C* is defined as floor( (RF / 2) = + 1 ), in case of RF=3D1 this in practice is the same as CL=3DONE.

On 10 Ma= rch 2015 at 18:10, DuyHai Doan <doanduyhai@gmail.com> wro= te:
First idea to elimin= ate any issue with regards to staled data: issue the same count query with = RF=3DQUORUM and check whether there are still inconsistencies

On Tue, Mar 10, 2015 at 9:13 AM, Rumph, Frens Jan <mail@fr= ensjan.nl> wrote:
Hi Jens, Mikhail, Dae= meon,

Thanks = for your replies. Sorry for my reply being late ... mails from the user-lis= t were moved to the wrong inbox on my side.

I'm in a development environment and thu= s using replication factor =3D 1 and consistency =3D ONE with three nodes. = So the 'results from different nodes between queries' hypothesis se= ems unlikely to me. I would expect a timeout if some node wouldn't be a= ble to answer.

I tried tracing, but I couldn't really make any of it.

For example I performed t= wo select distinct ... from ... queries: Traces for both of them contained = more than one line like 'Submitting range requests on ... ranges ...= 9; and 'Submitted ... concurrent range requests covering ... ranges'= ;. These lines occur with varying numbers, e.g. :

Submitting range r= equests on 593 ranges with a concurrency of 75 (1.35 rows per range expecte= d)=C2=A0
Submitting range requests on 769 ranges with a concurrency of 7= 5 (1.35 rows per range expected)

Also when looking= at the lines like 'Executing seq scan across ... sstables for ...'= I saw that in one case which yielded way less partition keys that only the= tokens from -9223372036854770000=C2=A0 to -594461978511041000 were include= d. In a case which yielded much more partition keys, the entire token range= did seem to be queried.
<= br style=3D"font-size:12.8000001907349px">To reiterate my initial questions: is this behavior to be expec= ted? Am I doing something wrong? Is there a workaround?

Best regards,
Frens Jan

On 4 March 2015 at 22:59, daemeon reiydelle <daemeo= nr@gmail.com> wrote:
What is the replication? Could you be servin= g stale data from a node that was not properly replicated (hints timeout ex= ceeded by a node being down?)



On Wed, Mar 4, 2015 at 11:03 AM, Jens Rantil= <jens.rantil@tink.se> wrote:
Frens,

What consistency are you querying with? Could be you are= simply receiving result from different nodes each time.

Jens

=E2=80=93
Skickat fr=C3=A5n Mailbox


On Wed, Mar 4, 2015 at 7:08 PM, Mikha= il Strebkov <strebkov@gmail.com> wrote:

We have observed the same issue in our production Cassandra clus= ter (5 nodes in one DC). We use Cassandra 2.1.3 (I joined the list too late= to realize we shouldn=E2=80=99t user 2.1.x yet) on Amazon machines (create= d from community AMI).

In addition to count variations with 5 to 10% we observe variations fo= r the query =E2=80=9Cselect * from table1 where time > '$fromDate= 9; and time < '$toDate' allow filtering=E2=80=9D results. We ite= rated through the results multiple times using official Java driver. We use= d that query for a huge data migration and were unpleasantly surprised that= it is unreliable. In our case =E2=80=9Cnodetool repair=E2=80=9D didn=E2=80= =99t fix the issue.

So I echo Frens questions.

Thanks,
Mikhail




On Wed, Mar 4, 2015 at 3:55 AM, Rumph, Frens Jan <<= a href=3D"mailto:mail@frensjan.nl" target=3D"_blank">mail@frensjan.nl&g= t; wrote:

Hi,

Is it to be expected that select count(*) from ... and select distinct= partition-key-columns from ... to yield inconsistent results between execu= tions even though the table at hand isn't written to?

I have a table in a keyspace with replication_factor =3D 1 which is so= mething like:

CREATE TABLE tbl (
=C2=A0 =C2=A0 id frozen<id_type>,
=C2=A0 =C2=A0 bucket bigint,
=C2=A0 =C2=A0 offset int,
=C2=A0 =C2=A0 value double,
=C2=A0 =C2=A0 PRIMARY KEY ((id, bucket), offset)
)

The frozen udt is:

CREATE TYPE id_type (
=C2=A0 =C2=A0 tags map<text, text>
);

When I do select count(*) from tbl several times the actual count vari= es with 5 to 10%. Also when performing select distinct id, bucket from tbl = the results aren't consistent over several query executions. The table = is not being written to at the time I performed the queries.

Is this to be expected? Or is this a bug? Is there a alternative metho= d / workaround?

I'm using cqlsh 5.0.1 with Cassandra 2.1.2 on 64bit fedora 21 with= Oracle Java 1.8.0_31.

Thanks in advance,
Frens Jan



<= /div>



--089e016338064585ad0510f8c1e7--