cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From graham sanderson <gra...@vast.com>
Subject Re: Cassandra write timeout
Date Mon, 29 Jun 2015 20:46:57 GMT
You should upgrade to 2.1.7 due to https://issues.apache.org/jira/browse/CASSANDRA-9549 <https://issues.apache.org/jira/browse/CASSANDRA-9549>

> On Jun 29, 2015, at 2:14 PM, Aditya Shetty <aditya.shetty@snapdeal.com> wrote:
> 
> Hi Carlos
> 
> I was mistaken earlier, I cross checked now, and found that  the DevOps team had installed
a later version, even though we asked for 2.1.3, the version installed on the server is 2.1.5.
> 
> Do you think upgrading to 2.1.6 would be useful?
> 
> On Tue, Jun 30, 2015 at 12:36 AM, Carlos Rolo <rolo@pythian.com <mailto:rolo@pythian.com>>
wrote:
> Hello,
> 
> Compaction issues are a know issue in the 2.1 series until 2.1.5.You can probably search
this mailing list for further details and/or similar situations. And 2.1.3 is not considered
a stable release. It is a good rule to always follow DSE releases as a guideline for which
version to pick. 
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com <http://www.pythian.com/>
> On Mon, Jun 29, 2015 at 8:53 PM, Aditya Shetty <aditya.shetty@snapdeal.com <mailto:aditya.shetty@snapdeal.com>>
wrote:
> Hi Carlos
> 
> Thanks for the tip. But is this a known issue in 2.1.3?
> 
> On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo <rolo@pythian.com <mailto:rolo@pythian.com>>
wrote:
> Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or 2.17 since it
is already the stable version). Given the size of your cluster should be a fairly easy operation
(Upgrade and rolling restart) for more information follow the documentation: http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
<http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html>
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 <tel:%2B1%20613%20565%208696%20x1649>
> www.pythian.com <http://www.pythian.com/>
> On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <aditya.shetty@snapdeal.com <mailto:aditya.shetty@snapdeal.com>>
wrote:
> The cassandra version Im using is : 2.1.3
> 
> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <aditya.shetty@snapdeal.com <mailto:aditya.shetty@snapdeal.com>>
wrote:
> Hi Carlos
> 
> The node with 5MB does have compactions running while the others don't:
> 
> pending tasks: 1
>    compaction type           keyspace          table   completed        total    unit
  progress
>         Compaction   reviews_platform   object_stats         295   1300550330   bytes
     0.00%
> Active compaction remaining time :   0h01m17s
> 
> 
> Also, the progress stays at 0% 
> 
> 
> On other nodes:
> pending tasks: 0
> 
> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <rolo@pythian.com <mailto:rolo@pythian.com>>
wrote:
> Hello,
> 
> Can you please tell what is the version you are running? What is the Replication Factor
and if the node with 5MB has compactions running (nodetool compactionstats)?
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 <tel:%2B1%20613%20565%208696%20x1649>
> www.pythian.com <http://www.pythian.com/>
> 
> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <aditya.shetty@snapdeal.com <mailto:aditya.shetty@snapdeal.com>>
wrote:
> Another interesting thing is that the node that has only 5MB if data (10.1.24.73) is
the one where CPU utilization is quite high (about 100%) while the other nodes are at <
5%
> 
> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <aditya.shetty@snapdeal.com <mailto:aditya.shetty@snapdeal.com>>
wrote:
> Hi
> 
> I have a 3 node cluster with a simple column family that stores product information.
However, even with low write loads, I get WriteTimeoutExceptions when writing into it.
> 
> Im doing around 10 writes per second.
> 
> Strangely, when I check the database, all the products that had timed out are actually
present in the database. I dont think this is because of hinted handoffs. I was monitoring
the cluster the whole time and none of the nodes went down.
> 
> Also, as per nodetool status the data is not equally distributed across all 3 nodes (even
though they are identical in terms of config)
> 
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address      Load       Tokens  Owns    Host ID                               Rack
> UN  10.1.16.131  48.57 MB   256     ?       5aee18b0-7a75-478a-8648-2f2b1c6821fd  rack1
> UN  10.1.18.107  54.72 MB   256     ?       c307526c-e362-4b61-abc4-a2e951e38b86  rack1
> UN  10.1.24.73   4.25 MB    256     ?       b10cb9ac-fbb1-4014-9534-26477d31fffd  rack1
> 
> 
> -- 
> 
> 
> Aditya Shetty
> Lead Engineer
> 
> 
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
> 
> 
> 	
> Download Our App
>  <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>

>  <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>

>  <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
> 
> 
> 
> -- 
> 
> 
> Aditya Shetty
> Lead Engineer
> 
> 
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
> 
> 
> 	
> Download Our App
>  <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>

>  <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>

>  <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
> 
> 
> --
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> 
> Aditya Shetty
> Lead Engineer
> 
> 
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
> 
> 
> 	
> Download Our App
>  <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>

>  <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>

>  <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
> 
> 
> 
> -- 
> 
> 
> Aditya Shetty
> Lead Engineer
> 
> 
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
> 
> 
> 	
> Download Our App
>  <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>

>  <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>

>  <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
> 
> 
> --
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> 
> Aditya Shetty
> Lead Engineer
> 
> 
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
> 
> 
> 	
> Download Our App
>  <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>

>  <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>

>  <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
> 
> 
> --
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> 
> Aditya Shetty
> Lead Engineer
> 
> 
> M: +91 7022423545, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
> 
> 
> 	
> Download Our App
>  <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>

>  <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>

>  <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>


Mime
View raw message