cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ZAIDI, ASAD A" <>
Subject RE: Node failure Due To Very high GC pause time
Date Mon, 03 Jul 2017 19:14:53 GMT
>>       here my is doubt is that does all the deleted 3.3Million row will be loaded
in my on-heap memory? if not what will be object that occupying those memory ?

          It depends on your queries what data they’re fetching from your database.   Assuming
you’re using CMS garbage collector and you’ve enabled GC logs with PrintGCDetails, PrintClassHistogramBeforeFullGC,
PrintClassHistogramAfterFullGC – your logs should tell you what java classes occupies most
of your  heap memory.

System.log file can also give you some clue  like if you see references to your tables with
[tombstones],  A quick [grep –i tombstone /path/to/system.log] command would tell you what
objects are suffering with tombstones!

From: Karthick V []
Sent: Monday, July 03, 2017 11:47 AM
To: user <>
Subject: Re: Node failure Due To Very high GC pause time

Hi Bryan,

            Thanks for your quick response.  We have already tuned our memory and GC based
on our hardware specification and it was working fine until yesterday, i.e before facing the
below specified delete request. As you specified we will once again look into our GC &
memory configuration.

FYKI :  We are using memtable_allocation_typ as offheap_objects.

Consider the following table

CREATE TABLE  EmployeeDetails (
    branch_id text,
    department_id  text,
    emp_id bigint,
    emp_details text,
    PRIMARY KEY (branch, department, emp_id)
) WITH CLUSTERING ORDER BY (department ASC, emp_id ASC)

    In this table I have 10 million records for the a particular branch_id and department_id
. And following are the list of operation which I perform in C* in chronological order

  1.  Deleting 5 million records, from the start, in batches of 500 records per request for
the particular branch_id (say 'xxx' ) and department_id (say 'yyy')
  2.  Read the next 500 records as soon the above delete operation is being completed ( Select
* from EmployeeDetails where branch_id='xxx' and department_id = 'yyy' and emp_id >50000000
limit 500 )

It's only after executing the above read request there was a spike in memory and within few
minutes the node has been marked down.

So my question here is , will the above read request will load all the deleted 5 million records
in my memory before it starts fetching or will it jump directly to the offset of 50000001
record (since we have specified the greater than condition) ? If its going to the former case
then for sure the read request will keep the data in main memory and performs merge operation
before it delivers the data as per this wiki(<>
). If not let me know how the above specified read request will provide me the data .

Note : And also while analyzing my heap dump its clear that majority of the memory is being
held my Tombstone threads.

Thanks in advance
-- karthick

---- On Mon, 03 Jul 2017 20:40:10 +0530 Bryan Cheng <<>>
wrote ----

This is a very antagonistic use case for Cassandra :P I assume you're familiar with Cassandra
and deletes? (eg.<>,<>)

That being said, are you giving enough time for your tables to flush to disk? Deletes generate
markers which can and will consume memory until they have a chance to be flushed, after which
they will impact query time and performance (but should relieve memory pressure). If you're
saturating the capability of your nodes your tables will have difficulty flushing. See<>.

This could also be a heap/memory configuration issue as well or a GC tuning issue (although
unlikely if you've left those at default)


On Mon, Jul 3, 2017 at 7:51 AM, Karthick V <<>>


      Recently In my test Cluster I faced a outrageous GC activity which made the Node unreachable
inside the cluster itself.

Scenario :
      In a Partition of 5Million rows we read first 500 (by giving the starting range) and
delete the same 500 again.The same has been done recursively by changing the Start range alone.
Initially I didn't see any difference in the query performance ( upto 50,000) but later I
observed a significant increase in performance when reached about a 3.3Million the read request
failed and the node went unreachable. After analysing my GC logs it is clear that 99% of my
old-memory space is occupied and there are no more space for allocation it caused the machine
       here my is doubt is that does all the deleted 3.3Million row will be loaded in my on-heap
memory? if not what will be object that occupying those memory ?.

PS : I am using C* 2.1.13 in cluster.

View raw message