cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Chan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5483) Repair tracing
Date Mon, 20 Oct 2014 23:12:36 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177629#comment-14177629
] 

Ben Chan commented on CASSANDRA-5483:
-------------------------------------

Some nice changes to the JMX interface in the last 3 months; I no longer have to add lots
of boilerplate function overloads.

{noformat}
read -r JMXGET <<E
/jmx_port/{p=\$2;} /binary/{split(\$2,a,/\047/);h=a[2];} \
END{printf("bin/nodetool -h %s -p %s\n",h,p,cmd);}
E

ccm_nodetool() { local N=$1; shift; $(ccm $N show | awk -F= "$JMXGET") "$@"; }

# git checkout 49833b9
W=https://issues.apache.org/jira/secure/attachment
for url in $W/12675963/5483-v17.patch; do
  { [ -e $(basename $url) ] || curl -sO $url; } && git apply $(basename $url)
done &&
ant clean && ant &&
./ccm-repair-test -kR &&
ccm node1 stop &&
ccm node1 clear &&
ccm node1 start &&
ccm_nodetool node1 repair -tr
{noformat}

----

Issues encountered during testing:
- My old method for forcing a streaming repair (yukim's "ccm clear" suggestion) no longer
seems to work. Which means I can no longer test tracing of streaming. Is there something else
I have to do now to force a streaming repair?
- 5483-16 and 5483-17 no longer filter traces based on traceType. Using my standard simple
test, I count 182 traces, versus 61 from a test run fom 2014-06, about 200% more traces (from
the query-tracing code paths), though I don't know how this percentage will change with longer
repairs. Some representative extraneous traces:

{noformat}
Sending message to /127.0.0.2
Sending message to /127.0.0.3
Message received from /127.0.0.3
Processing response from /127.0.0.3
Message received from /127.0.0.2
Processing response from /127.0.0.2

Parsing SELECT * FROM system.sstable_activity WHERE keyspace_name=? and columnfamily_name=?
and generation=?
Preparing statement
Executing single-partition query on sstable_activity
Acquiring sstable references
Merging memtable tombstones
Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones
{noformat}


> Repair tracing
> --------------
>
>                 Key: CASSANDRA-5483
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Yuki Morishita
>            Assignee: Ben Chan
>            Priority: Minor
>              Labels: repair
>             Fix For: 3.0
>
>         Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch,
5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, trunk@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
trunk@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
>
>
> I think it would be nice to log repair stats and results like query tracing stores traces
to system keyspace. With it, you don't have to lookup each log file to see what was the status
and how it performed the repair you invoked. Instead, you can query the repair log with session
ID to see the state and stats of all nodes involved in that repair session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message