cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wickman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-11577) Traces persist for longer than 24 hours
Date Thu, 14 Apr 2016 21:28:25 GMT
Josh Wickman created CASSANDRA-11577:
----------------------------------------

             Summary: Traces persist for longer than 24 hours
                 Key: CASSANDRA-11577
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11577
             Project: Cassandra
          Issue Type: Bug
            Reporter: Josh Wickman
            Priority: Minor


My deployment currently has clusters on both Cassandra 1.2 (1.2.19) and 2.1 (2.1.11) with
tracing on.  On 2.1, the trace records persist for longer than the [documented 24 hours|https://docs.datastax.com/en/cql/3.3/cql/cql_reference/tracing_r.html]:

{noformat}
cqlsh> select started_at from system_traces.sessions limit 10;

 started_at
--------------------------
 2016-03-11 23:28:40+0000
 2016-03-14 21:09:07+0000
 2016-03-14 16:42:25+0000
 2016-03-14 16:13:13+0000
 2016-03-14 19:12:11+0000
 2016-03-14 21:25:57+0000
 2016-03-29 22:45:28+0000
 2016-03-14 19:56:27+0000
 2016-03-09 23:31:41+0000
 2016-03-10 23:08:44+0000

(10 rows)
{noformat}

My systems on 1.2 do not exhibit this problem:

{noformat}
cqlsh> select started_at from system_traces.sessions limit 10;

 started_at
--------------------------
 2016-04-13 22:49:31+0000
 2016-04-14 18:06:45+0000
 2016-04-14 07:57:00+0000
 2016-04-14 04:35:05+0000
 2016-04-14 03:54:20+0000
 2016-04-14 10:54:38+0000
 2016-04-14 18:34:04+0000
 2016-04-14 12:56:57+0000
 2016-04-14 01:57:20+0000
 2016-04-13 21:36:01+0000
{noformat}

The event records also persist alongside the session records, for example:

{noformat}
cqlsh> select session_id, dateOf(event_id) from system_traces.events where session_id =
fc8c1e80-e7e0-11e5-a2fb-1968ff3c067b;

 session_id                           | dateOf(event_id)
--------------------------------------+--------------------------
 fc8c1e80-e7e0-11e5-a2fb-1968ff3c067b | 2016-03-11 23:28:40+0000
{noformat}

Between these versions, the table parameter {{default_time_to_live}} was introduced.  The
{{system_traces}} tables report the default value of 0:

{noformat}
cqlsh> desc table system_traces.sessions

CREATE TABLE system_traces.sessions (
    session_id uuid PRIMARY KEY,
    coordinator inet,
    duration int,
    parameters map<text, text>,
    request text,
    started_at timestamp
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = 'traced sessions'
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.SnappyCompressor'}
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 0
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';
{noformat}

I suspect that {{default_time_to_live}} is superseding the mechanism used in 1.2 to expire
the trace records.  Evidently I cannot change this parameter for this table:

{noformat}
cqlsh> alter table system_traces.sessions with default_time_to_live = 86400;
Unauthorized: code=2100 [Unauthorized] message="Cannot ALTER <table system_traces.sessions>"
{noformat}

I realize Cassandra 1.2 is no longer supported, but the problem is being manifested in Cassandra
2.1 for me (I included 1.2 only for comparison).  Since I couldn't find an existing ticket
addressing this issue, I'm concerned that it may be present in more recent versions of Cassandra
as well, but I have not tested these.

The persistent trace records are contributing to disk filling, and more importantly, making
it more difficult to analyze the trace data.  Is there a workaround for this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message