cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Kogan <pavel.ko...@cortica.com>
Subject Re: Batch of prepared statements exceeding specified threshold
Date Fri, 20 Jun 2014 17:51:10 GMT
Hi Marcelo,

No pending write tasks, I am writing a lot, about 100-200 writes each up to
100Kb every 15[s].
It is running on decent cluster of 5 identical nodes, quad cores i7 with
32Gb RAM and 480Gb SSD.

Regards,
  Pavel


On Fri, Jun 20, 2014 at 12:31 PM, Marcelo Elias Del Valle <
marcelo@s1mbi0se.com.br> wrote:

> Pavel,
>
> In my case, the heap was filling up faster than it was draining. I am
> still looking for the cause of it, as I could drain really fast with SSD.
>
> However, in your case you could check (AFAIK) nodetool tpstats and see if
> there are too many pending write tasks, for instance. Maybe you really are
> writting more than the nodes are able to flush to disk.
>
> How many writes per second are you achieving?
>
> Also, I would look for GCInspector in the log:
>
> cat system.log* | grep GCInspector | wc -l
> tail -1000 system.log | grep GCInspector
>
> Do you see it running a lot? Is it taking much more time to run each time
> it runs?
>
> I am no Cassandra expert, but I would try these things first and post the
> results here. Maybe other people in the list have more ideas.
>
> Best regards,
> Marcelo.
>
>
> 2014-06-20 8:50 GMT-03:00 Pavel Kogan <pavel.kogan@cortica.com>:
>
> The cluster is new, so no updates were done. Version 2.0.8.
>> It happened when I did many writes (no reads). Writes are done in small
>> batches of 2 inserts (writing to 2 column families). The values are big
>> blobs (up to 100Kb).
>>
>> Any clues?
>>
>> Pavel
>>
>>
>> On Thu, Jun 19, 2014 at 8:07 PM, Marcelo Elias Del Valle <
>> marcelo@s1mbi0se.com.br> wrote:
>>
>>> Pavel,
>>>
>>> Out of curiosity, did it start to happen before some update? Which
>>> version of Cassandra are you using?
>>>
>>> []s
>>>
>>>
>>> 2014-06-19 16:10 GMT-03:00 Pavel Kogan <pavel.kogan@cortica.com>:
>>>
>>>> What a coincidence! Today happened in my cluster of 7 nodes as well.
>>>>
>>>> Regards,
>>>>   Pavel
>>>>
>>>>
>>>> On Wed, Jun 18, 2014 at 11:13 AM, Marcelo Elias Del Valle <
>>>> marcelo@s1mbi0se.com.br> wrote:
>>>>
>>>>> I have a 10 node cluster with cassandra 2.0.8.
>>>>>
>>>>> I am taking this exceptions in the log when I run my code. What my
>>>>> code does is just reading data from a CF and in some cases it writes
new
>>>>> data.
>>>>>
>>>>>  WARN [Native-Transport-Requests:553] 2014-06-18 11:04:51,391
>>>>> BatchStatement.java (line 228) Batch of prepared statements for
>>>>> [identification1.entity, identification1.entity_lookup] is of size 6165,
>>>>> exceeding specified threshold of 5120 by 1045.
>>>>>  WARN [Native-Transport-Requests:583] 2014-06-18 11:05:01,152
>>>>> BatchStatement.java (line 228) Batch of prepared statements for
>>>>> [identification1.entity, identification1.entity_lookup] is of size 21266,
>>>>> exceeding specified threshold of 5120 by 16146.
>>>>>  WARN [Native-Transport-Requests:581] 2014-06-18 11:05:20,229
>>>>> BatchStatement.java (line 228) Batch of prepared statements for
>>>>> [identification1.entity, identification1.entity_lookup] is of size 22978,
>>>>> exceeding specified threshold of 5120 by 17858.
>>>>>  INFO [MemoryMeter:1] 2014-06-18 11:05:32,682 Memtable.java (line 481)
>>>>> CFS(Keyspace='OpsCenter', ColumnFamily='rollups300') liveRatio is
>>>>> 14.249755859375 (just-counted was 9.85302734375).  calculation took 3ms
for
>>>>> 1024 cells
>>>>>
>>>>> After some time, one node of the cluster goes down. Then it goes back
>>>>> after some seconds and another node goes down. It keeps happening and
there
>>>>> is always a node down in the cluster, when it goes back another one falls.
>>>>>
>>>>> The only exceptions I see in the log is "connected reset by the peer",
>>>>> which seems to be relative to gossip protocol, when a node goes down.
>>>>>
>>>>> Any hint of what could I do to investigate this problem further?
>>>>>
>>>>> Best regards,
>>>>> Marcelo Valle.
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message