cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Deng (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-11714) shutdown script should wait more than 30 seconds before force kill
Date Thu, 05 May 2016 00:59:12 GMT
Wei Deng created CASSANDRA-11714:
------------------------------------

             Summary: shutdown script should wait more than 30 seconds before force kill
                 Key: CASSANDRA-11714
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11714
             Project: Cassandra
          Issue Type: Improvement
          Components: Tools
            Reporter: Wei Deng
            Priority: Minor


I'm running with 3.0.6 and it appears that if I let it run for multiple days (ingesting a
never-ending streaming data from Kafka), then the next time I'm restarting the JVM by "service
cassandra restart", it will take 9 minutes to replay the commit log segments (the commitlog_total_space_in_mb
is using the default 8GB and I observed close to 8GB commitlog segment files before the restart).
Here is a fragment of the system.log showing how long it takes:

{noformat}
INFO  [main] 2016-05-04 19:06:54,356  CommitLog.java:168 - Replaying /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248187.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248191.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248195.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248198.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248201.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248205.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248209.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248213.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248217.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248221.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248225.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248230.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248234.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248238.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248242.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248246.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248250.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248254.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248258.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248262.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248266.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248270.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248274.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248278.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248282.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248286.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248290.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248294.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248299.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248303.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248307.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248311.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248315.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248319.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248323.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248327.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248331.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248335.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248340.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248344.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248348.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248352.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248356.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248360.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248364.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248368.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248372.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248376.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248380.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248384.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248388.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248392.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248396.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248397.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248401.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248405.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248409.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248413.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248417.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248421.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248425.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248429.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248433.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248437.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248441.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248445.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248449.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248453.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248457.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248461.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248466.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248470.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248474.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248478.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248482.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248486.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248490.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248494.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248498.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248502.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248506.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248510.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248514.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248518.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248522.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248526.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248528.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248529.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248530.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248531.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248532.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248533.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248534.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248535.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248536.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248537.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248538.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248539.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248540.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248541.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248542.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248543.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248544.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248545.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248546.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248547.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248548.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248549.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248550.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248551.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248552.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248553.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248554.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248555.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248556.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248557.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248558.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248559.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248560.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248561.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248562.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248563.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248564.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248565.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248566.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248567.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248568.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248569.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248570.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248571.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248572.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248573.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248574.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248575.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248576.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248577.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248578.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248579.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248580.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248581.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248582.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248583.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248584.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248585.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248586.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248587.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248588.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248589.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248590.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248591.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248592.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248593.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248594.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248595.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248596.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248597.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248598.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248599.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248600.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248601.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248602.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248603.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248604.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248605.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248606.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248607.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248608.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248609.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248610.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248611.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248612.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248613.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248614.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248615.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248616.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248617.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248618.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248619.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248620.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248621.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248622.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248623.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248624.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248625.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248626.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248627.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248628.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248629.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248630.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248631.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248632.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248633.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248634.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248635.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248636.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248637.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248638.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248639.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248640.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248641.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248642.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248643.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248644.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248645.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248646.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248647.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248648.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248649.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248650.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248651.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248652.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248653.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248654.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248655.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248656.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248657.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248658.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248659.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248660.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248661.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248662.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248663.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248664.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248665.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248666.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248667.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248668.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248669.log, /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248670.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248671.log
INFO  [main] 2016-05-04 19:15:44,172  CommitLog.java:170 - Log replay complete, 331463 replayed
mutations
{noformat}

It appears that the init script here only gives 30-second grace period after SIGTERM before
forcefully killing it.

https://github.com/apache/cassandra/blob/cassandra-3.0/debian/init/#L101

If we could increase this 30-second to a higher wait time like 1 or 2 minutes we could make
the startup delay much shorter.

Keep in mind that we don't always have to wait for this long during the shutdown so we're
not really losing anything with this improvement. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message