cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya Shetty <aditya.she...@snapdeal.com>
Subject Re: Cassandra write timeout
Date Mon, 29 Jun 2015 18:53:59 GMT
Hi Carlos

Thanks for the tip. But is this a known issue in 2.1.3?

On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo <rolo@pythian.com> wrote:

> Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or
> 2.17 since it is already the stable version). Given the size of your
> cluster should be a fairly easy operation (Upgrade and rolling restart) for
> more information follow the documentation:
> http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <aditya.shetty@snapdeal.com
> > wrote:
>
>> The cassandra version Im using is : 2.1.3
>>
>> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <
>> aditya.shetty@snapdeal.com> wrote:
>>
>>> Hi Carlos
>>>
>>> The node with 5MB does have compactions running while the others don't:
>>>
>>>
>>>
>>>
>>> *pending tasks: 1   compaction type           keyspace          table
>>> completed        total    unit   progress        Compaction
>>> reviews_platform   object_stats         295   1300550330   bytes
>>> 0.00%Active compaction remaining time :   0h01m17s*
>>>
>>>
>>> Also, the progress stays at 0%
>>>
>>>
>>> On other nodes:
>>> *pending tasks: 0*
>>>
>>> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <rolo@pythian.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Can you please tell what is the version you are running? What is the
>>>> Replication Factor and if the node with 5MB has compactions running
>>>> (nodetool compactionstats)?
>>>>
>>>> Regards,
>>>>
>>>> Carlos Juzarte Rolo
>>>> Cassandra Consultant
>>>>
>>>> Pythian - Love your data
>>>>
>>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>>> www.pythian.com
>>>>
>>>> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <
>>>> aditya.shetty@snapdeal.com> wrote:
>>>>
>>>>> Another interesting thing is that the node that has only 5MB if data
>>>>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>>>>> while the other nodes are at < 5%
>>>>>
>>>>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>>>>> aditya.shetty@snapdeal.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I have a 3 node cluster with a simple column family that stores
>>>>>> product information. However, even with low write loads, I get
>>>>>> WriteTimeoutExceptions when writing into it.
>>>>>>
>>>>>> Im doing around 10 writes per second.
>>>>>>
>>>>>> Strangely, when I check the database, all the products that had timed
>>>>>> out are actually present in the database. I dont think this is because
of
>>>>>> hinted handoffs. I was monitoring the cluster the whole time and
none of
>>>>>> the nodes went down.
>>>>>>
>>>>>> Also, as per nodetool status the data is not equally distributed
>>>>>> across all 3 nodes (even though they are identical in terms of config)
>>>>>>
>>>>>> Datacenter: datacenter1
>>>>>> =======================
>>>>>> Status=Up/Down
>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>> --  Address      Load       Tokens  Owns    Host
>>>>>> ID                               Rack
>>>>>> UN  10.1.16.131  48.57 MB   256     ?
>>>>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd  rack1
>>>>>> UN  10.1.18.107  54.72 MB   256     ?
>>>>>> c307526c-e362-4b61-abc4-a2e951e38b86  rack1
>>>>>> UN  10.1.24.73   4.25 MB    256     ?
>>>>>> b10cb9ac-fbb1-4014-9534-26477d31fffd  rack1
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Aditya Shetty
>>>>>> *Lead Engineer*
>>>>>>
>>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>>> 2nd FLOOR, WEST WING,
>>>>>>
>>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>>
>>>>>>    Download Our App   [image: A]
>>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>>> [image: A]
>>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>>> [image: W]
>>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Aditya Shetty
>>>>> *Lead Engineer*
>>>>>
>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>> 2nd FLOOR, WEST WING,
>>>>>
>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>
>>>>>    Download Our App   [image: A]
>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>> [image: A]
>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>> [image: W]
>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>>    Download Our App   [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>>    Download Our App   [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
> --
>
>
>
>


-- 

Aditya Shetty
*Lead Engineer*

*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,

SALARPURIA SUPREME, MARATHAHALLI, BENGALURU

   Download Our App   [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>

Mime
View raw message