From user-return-59851-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Mon Feb 19 10:31:44 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 19319180607 for ; Mon, 19 Feb 2018 10:31:42 +0100 (CET) Received: (qmail 22099 invoked by uid 500); 19 Feb 2018 09:31:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22089 invoked by uid 99); 19 Feb 2018 09:31:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Feb 2018 09:31:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B31D218028B for ; Mon, 19 Feb 2018 09:31:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.129 X-Spam-Level: ** X-Spam-Status: No, score=2.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id LmQnLkAKf3HH for ; Mon, 19 Feb 2018 09:31:38 +0000 (UTC) Received: from mail-pl0-f49.google.com (mail-pl0-f49.google.com [209.85.160.49]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 36E0C5F4DC for ; Mon, 19 Feb 2018 09:31:37 +0000 (UTC) Received: by mail-pl0-f49.google.com with SMTP id s13so5338520plq.6 for ; Mon, 19 Feb 2018 01:31:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=b+St6YdgebDJcPGkmdxhYTIS7cAz0Y1U9RjQ5fAFo4Q=; b=PU7NrbGz9IKDbvo9zm9Rv3IZLngolJLHqeat5DWVjgd3ZEB4+M/P0qDUOBNtrcF6SZ 8CdLjjbiwoA9UOhrya2dxGi2jDyZHUMu1+4otIDK5Iu39+CGY7GOTL3OGiCZQksI4yee MrwzKad2R6AxGMQUQklavJu3idf11nHT2aplSfJdPWghsvb8p2fRkFtcNKE1SQniSaal MNS/aXWzdJW2xfEj7cvsK6ob/oP4gYQgKgK7u+ng3SIj1lA+ruxE5+hP7msMf5vONj2i jsuOT4HRTna7ZzsNpXnPmp3ezv1/kpErWjsb+EzGCm43rrsYYZFJK0XF4MqR6Ps+bA/k 09Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=b+St6YdgebDJcPGkmdxhYTIS7cAz0Y1U9RjQ5fAFo4Q=; b=j6JqHT0ty2WaTDTp4mkfMHLUpSYbX/ThCG280RwKZLeIA2b0TKLOyEFz/LM4tUNA0I VmHfugPFWI7L4X4kmc44zx+EC9yJWPpwWJNfXPmnIuMwKjwcBAuv9NeM4NdA8C9V2sTd ZDveDO6QGJrpLauWGQky5yY2MS41cJh8ykJhTDNC4r1fFX/Xf+OYajeGF7NI7DmxYlxQ EYE+7ItIVnhHI2iMtP/JmVrsOBUG9N6wacs3RwA/fPtGao9biQowFqmTgSHnOqTwVTu2 /F1lKVV2ws4lUz/Nj8HgF7ZyqqXB6UTmj9u9r5cLfah9vVuDuT5vvqkIgDmMQiuXRaQx fi7Q== X-Gm-Message-State: APf1xPCfz125Qa61o6ajc4FnoQrKg/joVMAwz/qRD9SR342xY/9nyPem 62CPHAXxb+LyVkZcN7iiXCKkCR96fAQ1kwhzbcY= X-Google-Smtp-Source: AH8x224aDfg7X5s2HtblZlm5R/hGzta8aT27dj0d7BHqlBKtziJWWez46bzZUFr9h7ybM0xuR9RWyxRIPRW09ldH6u4= X-Received: by 2002:a17:902:981:: with SMTP id 1-v6mr13852024pln.345.1519032689290; Mon, 19 Feb 2018 01:31:29 -0800 (PST) MIME-Version: 1.0 Sender: jpare86@gmail.com Received: by 10.236.137.146 with HTTP; Mon, 19 Feb 2018 01:31:08 -0800 (PST) In-Reply-To: References: From: Javier Pareja Date: Mon, 19 Feb 2018 09:31:08 +0000 X-Google-Sender-Auth: A2VJUNwqc5B6jKP-elY7UdzR0EE Message-ID: Subject: Re: Cassandra counter readtimeout error To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000d8f42505658d5b22" --000000000000d8f42505658d5b22 Content-Type: text/plain; charset="UTF-8" Hi, Thank you for your reply. As I was bothered by this problem, last night I upgraded the cluster to version 3.11.1 and everything is working now. As far as I can tell the counter table can be read now. I will be doing more testing today with this version but it is looking good. To answer your questions: - I might not have explained the table definition very well but the table does not have 6 partitions, but 6 partition keys. There are thousands of partitions in that table, a combination of all those partition keys. I also made sure that the partitions remained small when designing the table. - I also enabled tracing in the CQLSH but it showed nothing when querying this row. It however did when querying other tables... Thanks again for your reply!! I am very excited to be part of the Cassandra user base. Javier F Javier Pareja On Mon, Feb 19, 2018 at 8:08 AM, Alain RODRIGUEZ wrote: > > Hello, > > This table has 6 partition keys, 4 primary keys and 5 counters. > > > I think the root issue is this ^. There might be some inefficiency or > issues with counter, but this design, makes Cassandra relatively > inefficient in most cases and using standard columns or counters > indifferently. > > Cassandra data is supposed to be well distributed for a maximal > efficiency. With only 6 partitions, if you have 6+ nodes, there is 100% > chances that the load is fairly imbalanced. If you have less nodes, it's > still probably poorly balanced. Also reading from a small number of > sstables and in parallel within many nodes ideally to split the work and > make queries efficient, but in this case cassandra is reading huge > partitions from one node most probably. When the size of the request is too > big it can timeout. I am not sure how pagination works with counters, but I > believe even if pagination is working, at some point, you are just reading > too much (or too inefficiently) and the timeout is reached. > > I imagined it worked well for a while as counters are very small columns / > tables compared to any event data but at some point you might have reached > 'physical' limit, because you are pulling *all* the information you need > from one partition (and probably many SSTables) > > Is there really no other way to design this use case? > > When data starts to be inserted, I can query the counters correctly from >> that particular row but after a few minutes updating the table with >> thousands of events, I get a read timeout every time >> > > Troubleshot: > - Use tracing to understand what takes so long with your queries > - Check for warns / error in the logs. Cassandra use to complain when it > is unhappy with the configurations. There a lot of interesting and it's > been a while I last had a failure with no relevant informations in the logs. > - Check SSTable per read and other read performances for this counter > table. Using some monitoring could make the reason of this timeout obvious. > If you use Datadog for example, I guess that a quick look at the "Read > Path" Dashboard would help. If you are using any other tool, look for > SSTable per reads, Tombstone scanned (if any), keycache hitrate, resources > (as maybe fast insert rate compactions and implicit 'read-before-writes' > are making the machine less responsive. > > Fix: > - Improve design to improve the findings you made above ^ > - Improve compaction strategy or read operations depending on the findings > above ^ > > I am not saying there is no bug in counters and in your version, but I > would say it is to early to state this, given the data model, some other > reasons could explain this slowness. > > If you don't have any monitoring in place, tracing and logs are a nice > place to start digging. If you want to share those here, we can help > interpreting outputs you will share if needed :). > > C*heers, > > Alain > > > 2018-02-17 11:40 GMT+00:00 Javier Pareja : > >> Hello everyone, >> >> I get a timeout error when reading a particular row from a large counters >> table. >> >> I have a storm topology that inserts data into a Cassandra counter table. >> This table has 6 partition keys, 4 primary keys and 5 counters. >> >> When data starts to be inserted, I can query the counters correctly from >> that particular row but after a few minutes updating the table with >> thousands of events, I get a readtimeout every time I try to read a >> particular row from the table (the most frequently updated). Other rows I >> can read quick and fine. Also if I run "select *", the top few hundreds are >> returned quick and fine as expected. The storm topology is stopped but the >> error is still there. >> >> I am using Cassandra 3.6. >> >> More information here: >> https://stackoverflow.com/q/48833146 >> >> Are counters in this version broken? I run the query from CQLSH and get >> the same error every time. I tried running it with trace enabled and get >> nothing but the error: >> >> ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} >> >> >> Any ideas? >> > > --000000000000d8f42505658d5b22 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Thank you for your reply.

As I was bothered by this problem, last night I upgraded th= e cluster to version 3.11.1 and everything is working now. As far as I can = tell the counter table can be read now. I will be doing more testing today = with this version but it is looking good.

To answe= r your questions:
- I might not have explained the table definiti= on very well but the table does not have 6 partitions, but 6 partition keys= . There are thousands of partitions in that table, a combination of all tho= se partition keys. I also made sure that the partitions remained small when= designing the table.
- I also enabled tracing in the CQLSH but i= t showed nothing when querying this row. It however did when querying other= tables...

Thanks again for your reply!! I am = very excited to be part of the Cassandra user base.

Javier



F Javier Pareja

On Mon, Feb 19, 2018 at 8:08 AM, Alain RODRI= GUEZ <arodrime@gmail.com> wrote:

Hello,

This table has 6 partition keys, 4 = primary keys and 5 counters.

= I think the root issue is this ^. There might be some inefficiency or issue= s with counter, but this design, makes Cassandra relatively inefficient in = most cases and using standard columns or counters indifferently.
=
Cassandra data is supposed to be well distributed for a maxi= mal efficiency. With only 6 partitions, if you have 6+ nodes, there is 100%= chances that the load is fairly imbalanced. If you have less nodes, it'= ;s still probably poorly balanced. Also reading from a small number of ssta= bles and in parallel within many nodes ideally to split the work and make q= ueries efficient, but in this case cassandra is reading huge partitions fro= m one node most probably. When the size of the request is too big it can ti= meout. I am not sure how pagination works with counters, but I believe even= if pagination is working, at some point, you are just reading too much (or= too inefficiently) and the timeout is reached.

I = imagined it worked well for a while as counters are very small columns / ta= bles compared to any event data but at some point you might have reached &#= 39;physical' limit, because you are pulling all the information = you need from one partition (and probably many SSTables)

Is t= here really no other way to design this use case?

When data starts to be i= nserted, I can query the counters correctly from that particular row but af= ter a few minutes updating the table with thousands of events, I get a read= timeout every time=C2=A0

Troubleshot:
- Use tracing to u= nderstand what takes so long with your queries
- Check for warns / error= in the logs. Cassandra use to complain when it is unhappy with the configu= rations. There a lot of interesting and it's been a while I last had a = failure with no relevant informations in the logs.
- Check SSTabl= e per read and other read performances for this counter table. Using some m= onitoring could make the reason of this timeout obvious. If you use Datadog= for example, I guess that a quick look at the "Read Path" Dashbo= ard would help. If you are using any other tool, look for SSTable per reads= , Tombstone scanned (if any), keycache hitrate, resources (as maybe fast in= sert rate compactions and implicit 'read-before-writes' are making = the machine less responsive.

Fix:
- Improve design to impr= ove the findings you made above ^
- Improve compaction strategy o= r read operations depending on the findings above ^

I am not saying there is no bug in counters and in your version, but I wo= uld say it is to early to state this, given the data model, some other reas= ons could explain this slowness.

If you don't = have any monitoring in place, tracing and logs are a nice place to start di= gging. If you want to share those here, we can help interpreting outputs yo= u will share if needed :).=C2=A0

C*heers,

Alain


2018-02-17 11:40 GMT+00:00= Javier Pareja <pareja.javier@gmail.com>:
Hello everyone,
I get a timeout error when reading a particular r= ow from a large counters table.

I have a storm topology that inserts data into a Cassa= ndra counter table. This table has 6 partition keys, 4 primary keys and 5 c= ounters.

When data starts to be inserted, I can query the cou= nters correctly from that particular row but after a few minutes updating t= he table with thousands of events, I get a readtimeout every time I try to = read a particular row from the table (the most frequently updated). Other r= ows I can read quick and fine. Also if I run "select *", the top = few hundreds are returned quick and fine as expected. The storm topology is= stopped but the error is still there.

I am using Cassandra 3= .6.

More information here:


Are counters in this version broken? I run the query from CQLSH = and get the same error every time. I tried running it with trace enabled an= d get nothing but the error:

ReadTimeout: Error from server: code=3D1200 [Coord=
inator node timed out waiting for replica nodes' responses] message=3D&=
quot;Operation timed out - received only 0 responses." info=3D{'re=
ceived_responses': 0, 'required_responses': 1, 'consistency=
': 'ONE'}

Any ideas?


--000000000000d8f42505658d5b22--