cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaydeep Chovatia <chovatia.jayd...@gmail.com>
Subject Re: Write timeout under load but Read is fine
Date Fri, 06 Mar 2015 18:12:05 GMT
I am using QUORUM
CQL
No SSDs (Anyway my IOPs is quite low so I dont think so it matters)
No compaction is running when I receive timeout


On Fri, Mar 6, 2015 at 12:35 AM, Carlos Rolo <rolo@pythian.com> wrote:

> What is the consistency level you are using?
> Are you using Thrift or CQL?
> Are you using SSDs?
> Check if compactions are running when you get the timeouts.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Tel: 1649
> www.pythian.com
>
> On Thu, Mar 5, 2015 at 9:51 PM, Jan <cnet62@yahoo.com> wrote:
>
>> Hello Jaydeep;
>>
>> Run *cassandra-stress* with R/W options enabled  for about the same time
>> and check if you have dropped packets.
>> It would eliminate the client as the source of the error & also give you
>> a replicable tool to base subsequent tests/ findings.
>>
>> Jan/
>>
>>
>>
>>
>>   On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>
>> I have tried increasing timeout to 10000 but no help. Also verified that
>> there is no network lost packets.
>>
>> Jaydeep
>>
>> On Wed, Mar 4, 2015 at 12:19 PM, Jan <cnet62@yahoo.com> wrote:
>>
>> HI Jaydeep;
>>
>>
>>    - look at the i/o  on all three nodes
>>    - Increase the write_request_timeout_in_ms: 10000
>>    - check the time-outs if any on the client inserting the Writes
>>    - check the Network for  dropped/lost packets
>>
>>
>> hope this helps
>> Jan/
>>
>>
>>
>>   On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>
>> Hi,
>>
>> In my test program when I increase load then I keep getting few "write
>> timeout" from Cassandra say every 10~15 mins. My read:write ratio is
>> 50:50. My reads are fine but only writes time out.
>>
>> Here is my Cassandra details:
>> Version: 2.0.11
>> Ring of 3 nodes with RF=3
>> Node configuration: 24 core + 64GB RAM + 2TB
>>
>> "write_request_timeout_in_ms: 5000", rest of Cassandra.yaml configuration
>> is default
>>
>> I've also checked IO on Cassandra nodes and looks very low (around 5%).
>> I've also checked Cassandra log file and do not see any GC happening. Also
>> CPU on Cassandra is low (around 20%). I have 20GB data on each node.
>>
>> My test program creates connection to all three Cassandra nodes and sends
>> read+write request randomly.
>>
>> Any idea what should I look for?
>>
>> Jaydeep
>>
>>
>>
>>
>>
>> --
>> Jaydeep
>>
>>
>>
>
> --
>
>
>
>


-- 
Jaydeep

Mime
View raw message