Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A33AD1796B for ; Mon, 29 Jun 2015 20:47:34 +0000 (UTC) Received: (qmail 3341 invoked by uid 500); 29 Jun 2015 20:47:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 3295 invoked by uid 500); 29 Jun 2015 20:47:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 3285 invoked by uid 99); 29 Jun 2015 20:47:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jun 2015 20:47:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of graham@vast.com designates 209.85.212.177 as permitted sender) Received: from [209.85.212.177] (HELO mail-wi0-f177.google.com) (209.85.212.177) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jun 2015 20:45:16 +0000 Received: by wiar9 with SMTP id r9so15864385wia.1 for ; Mon, 29 Jun 2015 13:47:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=qJRKdOSZZpoZ6AmZtSgK4J2BWVhT8LaAfLvEZhGdskA=; b=icAL0E4Ylew7qAofmkXTeGiNYcV5KJhkfpKfbspbWMWOk+z3Ee6Tpxmqreu3l60nd2 eCKeY49ePrTwGlBheC5AVvQGUFmid4uUKRO/KAqtl5cGnWbn+m2Eom2xaR0L74rDpyt1 eb8bdbvlT2qPfN1MkLnXp2r0ZR2LVIMofh0lFzA2qmQOpdvFKS46f8/cXb9H6l2h8TUz 2MvbvO9bUpTCuZzI12tfIWel/fLEdvbqpgDRuDC6T1wc1OGWtUEqX0mT2hhKlfj1u5z7 yGuZkZm5L9rgKFviNE6hKE5ILY7dhA2q/IU9jynDGF5ZV4yABO+VGCxbaIKN9X3EvdlV ozDA== X-Gm-Message-State: ALoCoQlIjTs4/JJZvTB68PHc27FMbA5Onp4uE0+IvyIY8MPs63eouA28TzJCvd+7aXyulD72Zwip X-Received: by 10.180.99.39 with SMTP id en7mr26538954wib.31.1435610824048; Mon, 29 Jun 2015 13:47:04 -0700 (PDT) Received: from [192.168.1.116] (cpe-70-113-52-246.austin.res.rr.com. [70.113.52.246]) by mx.google.com with ESMTPSA id c11sm13822682wib.1.2015.06.29.13.47.02 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 29 Jun 2015 13:47:03 -0700 (PDT) From: graham sanderson Content-Type: multipart/signed; boundary="Apple-Mail=_3EE3B844-0026-4F75-A105-D14BA51EF81C"; protocol="application/pkcs7-signature"; micalg=sha1 Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: Cassandra write timeout Date: Mon, 29 Jun 2015 15:46:57 -0500 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.2098) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_3EE3B844-0026-4F75-A105-D14BA51EF81C Content-Type: multipart/alternative; boundary="Apple-Mail=_86EDE64C-E441-4154-8BAD-F263B966C9DB" --Apple-Mail=_86EDE64C-E441-4154-8BAD-F263B966C9DB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii You should upgrade to 2.1.7 due to = https://issues.apache.org/jira/browse/CASSANDRA-9549 = > On Jun 29, 2015, at 2:14 PM, Aditya Shetty = wrote: >=20 > Hi Carlos >=20 > I was mistaken earlier, I cross checked now, and found that the = DevOps team had installed a later version, even though we asked for = 2.1.3, the version installed on the server is 2.1.5. >=20 > Do you think upgrading to 2.1.6 would be useful? >=20 > On Tue, Jun 30, 2015 at 12:36 AM, Carlos Rolo > wrote: > Hello, >=20 > Compaction issues are a know issue in the 2.1 series until 2.1.5.You = can probably search this mailing list for further details and/or similar = situations. And 2.1.3 is not considered a stable release. It is a good = rule to always follow DSE releases as a guideline for which version to = pick.=20 >=20 > Regards, >=20 > Carlos Juzarte Rolo > Cassandra Consultant > =20 > Pythian - Love your data >=20 > rolo@pythian | Twitter: cjrolo | Linkedin: = linkedin.com/in/carlosjuzarterolo = > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 > www.pythian.com > On Mon, Jun 29, 2015 at 8:53 PM, Aditya Shetty = > wrote: > Hi Carlos >=20 > Thanks for the tip. But is this a known issue in 2.1.3? >=20 > On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo > wrote: > Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or = 2.17 since it is already the stable version). Given the size of your = cluster should be a fairly easy operation (Upgrade and rolling restart) = for more information follow the documentation: = http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandra= Details.html = >=20 > Regards, >=20 > Carlos Juzarte Rolo > Cassandra Consultant > =20 > Pythian - Love your data >=20 > rolo@pythian | Twitter: cjrolo | Linkedin: = linkedin.com/in/carlosjuzarterolo = > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 = > www.pythian.com > On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty = > wrote: > The cassandra version Im using is : 2.1.3 >=20 > On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty = > wrote: > Hi Carlos >=20 > The node with 5MB does have compactions running while the others = don't: >=20 > pending tasks: 1 > compaction type keyspace table completed = total unit progress > Compaction reviews_platform object_stats 295 = 1300550330 bytes 0.00% > Active compaction remaining time : 0h01m17s >=20 >=20 > Also, the progress stays at 0%=20 >=20 >=20 > On other nodes: > pending tasks: 0 >=20 > On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo > wrote: > Hello, >=20 > Can you please tell what is the version you are running? What is the = Replication Factor and if the node with 5MB has compactions running = (nodetool compactionstats)? >=20 > Regards, >=20 > Carlos Juzarte Rolo > Cassandra Consultant > =20 > Pythian - Love your data >=20 > rolo@pythian | Twitter: cjrolo | Linkedin: = linkedin.com/in/carlosjuzarterolo = > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 = > www.pythian.com >=20 > On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty = > wrote: > Another interesting thing is that the node that has only 5MB if data = (10.1.24.73) is the one where CPU utilization is quite high (about 100%) = while the other nodes are at < 5% >=20 > On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty = > wrote: > Hi >=20 > I have a 3 node cluster with a simple column family that stores = product information. However, even with low write loads, I get = WriteTimeoutExceptions when writing into it. >=20 > Im doing around 10 writes per second. >=20 > Strangely, when I check the database, all the products that had timed = out are actually present in the database. I dont think this is because = of hinted handoffs. I was monitoring the cluster the whole time and none = of the nodes went down. >=20 > Also, as per nodetool status the data is not equally distributed = across all 3 nodes (even though they are identical in terms of config) >=20 > Datacenter: datacenter1 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Status=3DUp/Down > |/ State=3DNormal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID = Rack > UN 10.1.16.131 48.57 MB 256 ? = 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1 > UN 10.1.18.107 54.72 MB 256 ? = c307526c-e362-4b61-abc4-a2e951e38b86 rack1 > UN 10.1.24.73 4.25 MB 256 ? = b10cb9ac-fbb1-4014-9534-26477d31fffd rack1 >=20 >=20 > --=20 >=20 >=20 > Aditya Shetty > Lead Engineer >=20 >=20 > M: +91 7022423545 , T: 080 46603000 EXT: 4417 > 2nd FLOOR, WEST WING, > SALARPURIA SUPREME, MARATHAHALLI, BENGALURU >=20 >=20 > =09 > Download Our App > = =09 > = =09 > = >=20 >=20 >=20 > --=20 >=20 >=20 > Aditya Shetty > Lead Engineer >=20 >=20 > M: +91 7022423545 , T: 080 46603000 EXT: 4417 > 2nd FLOOR, WEST WING, > SALARPURIA SUPREME, MARATHAHALLI, BENGALURU >=20 >=20 > =09 > Download Our App > = =09 > = =09 > = >=20 >=20 > -- >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > --=20 >=20 >=20 > Aditya Shetty > Lead Engineer >=20 >=20 > M: +91 7022423545 , T: 080 46603000 EXT: 4417 > 2nd FLOOR, WEST WING, > SALARPURIA SUPREME, MARATHAHALLI, BENGALURU >=20 >=20 > =09 > Download Our App > = =09 > = =09 > = >=20 >=20 >=20 > --=20 >=20 >=20 > Aditya Shetty > Lead Engineer >=20 >=20 > M: +91 7022423545 , T: 080 46603000 EXT: 4417 > 2nd FLOOR, WEST WING, > SALARPURIA SUPREME, MARATHAHALLI, BENGALURU >=20 >=20 > =09 > Download Our App > = =09 > = =09 > = >=20 >=20 > -- >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > --=20 >=20 >=20 > Aditya Shetty > Lead Engineer >=20 >=20 > M: +91 7022423545 , T: 080 46603000 EXT: 4417 > 2nd FLOOR, WEST WING, > SALARPURIA SUPREME, MARATHAHALLI, BENGALURU >=20 >=20 > =09 > Download Our App > = =09 > = =09 > = >=20 >=20 > -- >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > --=20 >=20 >=20 > Aditya Shetty > Lead Engineer >=20 >=20 > M: +91 7022423545, T: 080 46603000 EXT: 4417 > 2nd FLOOR, WEST WING, > SALARPURIA SUPREME, MARATHAHALLI, BENGALURU >=20 >=20 > =09 > Download Our App > = =09 > = =09 > = --Apple-Mail=_86EDE64C-E441-4154-8BAD-F263B966C9DB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii You should upgrade to 2.1.7 due to https://issues.apache.org/jira/browse/CASSANDRA-9549

On Jun 29, 2015, at 2:14 PM, Aditya Shetty = <aditya.shetty@snapdeal.com> wrote:

Hi Carlos

I was mistaken earlier, I cross checked now, and found = that  the DevOps team had installed a later version, even though we = asked for 2.1.3, the version installed on the server is 2.1.5.

Do you think upgrading to 2.1.6 would be = useful?

On Tue, Jun 30, 2015 at 12:36 AM, Carlos Rolo = <rolo@pythian.com> wrote:
Hello,

Compaction issues are a know issue in the 2.1 series = until 2.1.5.You can probably search this mailing list for further = details and/or similar situations. And 2.1.3 is not considered a stable = release. It is a good rule to always follow DSE releases as a guideline = for which version to pick.

Regards,

Carlos Juzarte Rolo
Cassandra = Consultant
 
Pythian - = Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 = x1649
www.pythian.com

On Mon, Jun 29, 2015 at 8:53 PM, Aditya Shetty = <aditya.shetty@snapdeal.com> wrote:
Hi Carlos

Thanks for the tip. But is this a known issue in = 2.1.3?

On Tue, = Jun 30, 2015 at 12:20 AM, Carlos Rolo <rolo@pythian.com> wrote:
Ok, as a first step I would recommend an upgrade to at least = 2.1.6 (Or 2.17 since it is already the stable version). Given the size = of your cluster should be a fairly easy operation (Upgrade and rolling = restart) for more information follow the documentation: http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgra= deCassandraDetails.html

Regards,

Carlos Juzarte Rolo
Cassandra = Consultant
 
Pythian - = Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty = <aditya.shetty@snapdeal.com> wrote:
The cassandra version Im using is : 2.1.3

On Tue, = Jun 30, 2015 at 12:06 AM, Aditya Shetty <aditya.shetty@snapdeal.com> = wrote:
Hi Carlos

The node with 5MB does have compactions = running while the others don't:

pending tasks: 1
   compaction = type           = keyspace          = table   completed        = total    unit   progress
        = Compaction   reviews_platform   = object_stats         = 295   1300550330   = bytes      0.00%
Active = compaction remaining time :   0h01m17s


Also, the progress stays at 0%


On other nodes:
pending tasks: 0

On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo = <rolo@pythian.com> wrote:
Hello,

Can = you please tell what is the version you are running? What is the = Replication Factor and if the node with 5MB has compactions running = (nodetool compactionstats)?

Regards,

Carlos Juzarte = Rolo
Cassandra Consultant
 
Pythian - Love your = data

rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Mon, Jun 29, 2015 at 8:20 = PM, Aditya Shetty <aditya.shetty@snapdeal.com> wrote:
Another interesting thing is that the node that has only 5MB = if data (10.1.24.73) is the one where CPU utilization is quite high = (about 100%) while the other nodes are at < 5%

On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty = <aditya.shetty@snapdeal.com> wrote:
Hi

I have a 3 node cluster = with a simple column family that stores product information. However, = even with low write loads, I get WriteTimeoutExceptions when writing = into it.

Im doing around 10 writes = per second.

Strangely, when I check = the database, all the products that had timed out are actually present = in the database. I dont think this is because of hinted handoffs. I was = monitoring the cluster the whole time and none of the nodes went = down.

Also, as per nodetool status = the data is not equally distributed across all 3 nodes (even though they = are identical in terms of config)

Datacenter:= datacenter1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down
|/ = State=3DNormal/Leaving/Joining/Moving
--  = Address      = Load       Tokens  = Owns    Host = ID            =             &n= bsp;      Rack
UN  = 10.1.16.131  48.57 MB   256     = ?       = 5aee18b0-7a75-478a-8648-2f2b1c6821fd  rack1
UN  = 10.1.18.107  54.72 MB   256     = ?       = c307526c-e362-4b61-abc4-a2e951e38b86  rack1
UN  = 10.1.24.73   4.25 MB    = 256     ?       = b10cb9ac-fbb1-4014-9534-26477d31fffd  rack1


--
=20

Aditya Shetty
Lead = Engineer

M: +91 7022423545, T: 080 46603000 EXT: 4417
 2nd FLOOR, = WEST WING,

SALARPURIA SUPREME, MARATHAHALLI, = BENGALURU

Download Our App
=
3D"
3D"
 3D"



--
=20

Aditya Shetty
Lead = Engineer

M: +91 7022423545, T: = 080 46603000 EXT: 4417
 2nd FLOOR, WEST = WING,

SALARPURIA SUPREME, MARATHAHALLI, = BENGALURU

Download Our = App
= =
3D"
 3D"
3D"


--







--
=20

Aditya Shetty
Lead = Engineer

M: +91 7022423545, T: = 080 46603000 EXT: 4417
 2nd FLOOR, WEST = WING,

SALARPURIA SUPREME, MARATHAHALLI, = BENGALURU

Download Our = App
= =
3D"
 3D"
3D"



--
=20

Aditya Shetty
Lead = Engineer

M: +91 7022423545, T: = 080 46603000 EXT: 4417
 2nd FLOOR, WEST = WING,

SALARPURIA SUPREME, MARATHAHALLI, = BENGALURU

Download Our = App
= =
3D"
 3D"
3D"


--







--
=20

Aditya Shetty
Lead = Engineer

M: +91 7022423545, T: = 080 46603000 EXT: 4417
 2nd FLOOR, WEST = WING,

SALARPURIA SUPREME, MARATHAHALLI, = BENGALURU

Download Our = App
= =
3D"
 3D"
3D"


--







--
=20

Aditya Shetty
Lead = Engineer

M: +91 7022423545, T: 080 46603000 EXT: 4417
 2nd FLOOR, WEST = WING,

SALARPURIA SUPREME, MARATHAHALLI, = BENGALURU

Download Our = App
= =
3D"
 3D"
3D"

= --Apple-Mail=_86EDE64C-E441-4154-8BAD-F263B966C9DB-- --Apple-Mail=_3EE3B844-0026-4F75-A105-D14BA51EF81C Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIICuzCCArcw ggIgAgIBTDANBgkqhkiG9w0BAQUFADCBojELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAk9SMREwDwYD VQQHEwhQb3J0bGFuZDEWMBQGA1UEChMNT21uaS1FeHBsb3JlcjEWMBQGA1UECxMNSVQgRGVwYXJ0 bWVudDEbMBkGA1UEAxMSd3d3LmNvcm5lcmNhc2UuY29tMSYwJAYJKoZIhvcNAQkBFhdibG9ja291 dEBjb3JuZXJjYXNlLmNvbTAeFw0xMTA0MDYxNjE0MzFaFw0yMTA0MDMxNjE0MzFaMIGjMQswCQYD VQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNjbzEWMBQG A1UEChMNVmFzdC5jb20gSW5jLjEUMBIGA1UECxMLRW5naW5lZXJpbmcxGTAXBgNVBAMTEEdyYWhh bSBTYW5kZXJzb24xHjAcBgkqhkiG9w0BCQEWD2dyYWhhbUB2YXN0LmNvbTCBnzANBgkqhkiG9w0B AQEFAAOBjQAwgYkCgYEAm4K/W/0VdaOiS6tC1G8tSCAw989XCsJXxVPiny/hND6T0jVv4vP0JRiO vNzH6uoINoKQfgUKa+GCqILdY7Jdx61/WKqxltFTu5D0H8sFFNIKgf9cd3yU6t2susKrxaDXRCul pmcJ3AFg4xuG3ZUZt+XTYhBebQfjwgGQh3/pkQUCAwEAATANBgkqhkiG9w0BAQUFAAOBgQCKW+hQ JqNkPRht5fl8FHku80BLAH9ezEJtZJ6EU9fcK9jNPkAJgSEgPXQ++jE+4iYI2nIb/h5RILUxd1Ht m/yZkNRUVCg0+0Qj6aMT/hfOT0kdP8/9OnbmIp2T6qvNN2rAGU58tt3cbuT2j3LMTS2VOGykK4He iNYYqr+K6sPDHTGCAy0wggMpAgEBMIGpMIGiMQswCQYDVQQGEwJVUzELMAkGA1UECBMCT1IxETAP BgNVBAcTCFBvcnRsYW5kMRYwFAYDVQQKEw1PbW5pLUV4cGxvcmVyMRYwFAYDVQQLEw1JVCBEZXBh cnRtZW50MRswGQYDVQQDExJ3d3cuY29ybmVyY2FzZS5jb20xJjAkBgkqhkiG9w0BCQEWF2Jsb2Nr b3V0QGNvcm5lcmNhc2UuY29tAgIBTDAJBgUrDgMCGgUAoIIB2TAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTA2MjkyMDQ2NThaMCMGCSqGSIb3DQEJBDEWBBRNYf8X SRpUTFfQ7Sr6sYY9Xk1oazCBugYJKwYBBAGCNxAEMYGsMIGpMIGiMQswCQYDVQQGEwJVUzELMAkG A1UECBMCT1IxETAPBgNVBAcTCFBvcnRsYW5kMRYwFAYDVQQKEw1PbW5pLUV4cGxvcmVyMRYwFAYD VQQLEw1JVCBEZXBhcnRtZW50MRswGQYDVQQDExJ3d3cuY29ybmVyY2FzZS5jb20xJjAkBgkqhkiG 9w0BCQEWF2Jsb2Nrb3V0QGNvcm5lcmNhc2UuY29tAgIBTDCBvAYLKoZIhvcNAQkQAgsxgayggakw gaIxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJPUjERMA8GA1UEBxMIUG9ydGxhbmQxFjAUBgNVBAoT DU9tbmktRXhwbG9yZXIxFjAUBgNVBAsTDUlUIERlcGFydG1lbnQxGzAZBgNVBAMTEnd3dy5jb3Ju ZXJjYXNlLmNvbTEmMCQGCSqGSIb3DQEJARYXYmxvY2tvdXRAY29ybmVyY2FzZS5jb20CAgFMMA0G CSqGSIb3DQEBAQUABIGAJNyf4d0XvIWOeenOTcJmTVHeluvRIeTmJgYbHH3XIgVWyDdLmMG9dgK9 gJnFgFL1BCuuiG46W4W7k7ZdzlefdH++6fTvf5Pnr4aHfS2Qm5+IstITPu/qTN5cr4qweSuv3+Wl bcJc8SUI3ojQvDIzmLUkis888Wu1aWdFOv49LvUAAAAAAAA= --Apple-Mail=_3EE3B844-0026-4F75-A105-D14BA51EF81C--