Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of chris.burroughs@gmail.com
 designates 209.85.213.44 as permitted sender)
Message-ID: <52953A97.4020002@gmail.com>
Date: Tue, 26 Nov 2013 19:19:35 -0500
From: Chris Burroughs <chris.burroughs@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20131013 Icedove/17.0.9
MIME-Version: 1.0
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: MiscStage Backup
Content-Type: multipart/mixed;
 boundary="------------000504070701010101040304"

This is a multi-part message in MIME format.
--------------000504070701010101040304
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

I'm trying to debug a node that has a backup in MiscStage.  Starting a 
bit under 24 hours ago the number of Pending tasks jumped to a bit under 
400 and hovered around there.  It looks like repair requests from other 
nodes  (tpstats on this node shows AntiEntropySessions: 0, 0, 0, which I 
think indicates it did not originate the repair).  After each MiscStage 
task completes a series of Streams are kicked off.

I am confused why MiscStage is backing up:
  (A) This node has only been down a few hours over the past week so it 
should not be wildly out of sync
  (B) no other node in this cluster has had a comparable backup of 
pending Misc stages.

Repairs are run on all nodes once a week.  Physical resources on this 
node are not particularity saturated compared to the rest of the 
cluster; reads are slower but I can't tell cause from effect in that case.

Graph of MiscStage pending tasks: http://imgur.com/sHqHTvt

This is with a 1.2.11-ish dual-DC vnode cluster.

--------------000504070701010101040304
Content-Type: text/plain; charset=UTF-8;
 name="misc.stack"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="misc.stack"

Ik1pc2NTdGFnZToxIiBkYWVtb24gcHJpbz0xMCB0aWQ9MHgwMDAwN2Y4NGU4NTk4ODAwIG5p
ZD0weDQzYjIgd2FpdGluZyBvbiBjb25kaXRpb24gWzB4MDAwMDdmODNjMzczNDAwMF0KICAg
amF2YS5sYW5nLlRocmVhZC5TdGF0ZTogV0FJVElORyAocGFya2luZykKICAgICAgICBhdCBz
dW4ubWlzYy5VbnNhZmUucGFyayhOYXRpdmUgTWV0aG9kKQogICAgICAgIC0gcGFya2luZyB0
byB3YWl0IGZvciAgPDB4MDAwMDAwMDY5ZDIzYzcwMD4gKGEgamF2YS51dGlsLmNvbmN1cnJl
bnQuRnV0dXJlVGFzayRTeW5jKQogICAgICAgIGF0IGphdmEudXRpbC5jb25jdXJyZW50Lmxv
Y2tzLkxvY2tTdXBwb3J0LnBhcmsoTG9ja1N1cHBvcnQuamF2YToxNTYpCiAgICAgICAgYXQg
amF2YS51dGlsLmNvbmN1cnJlbnQubG9ja3MuQWJzdHJhY3RRdWV1ZWRTeW5jaHJvbml6ZXIu
cGFya0FuZENoZWNrSW50ZXJydXB0KEFic3RyYWN0UXVldWVkU3luY2hyb25pemVyLmphdmE6
ODExKQogICAgICAgIGF0IGphdmEudXRpbC5jb25jdXJyZW50LmxvY2tzLkFic3RyYWN0UXVl
dWVkU3luY2hyb25pemVyLmRvQWNxdWlyZVNoYXJlZEludGVycnVwdGlibHkoQWJzdHJhY3RR
dWV1ZWRTeW5jaHJvbml6ZXIuamF2YTo5NjkpCiAgICAgICAgYXQgamF2YS51dGlsLmNvbmN1
cnJlbnQubG9ja3MuQWJzdHJhY3RRdWV1ZWRTeW5jaHJvbml6ZXIuYWNxdWlyZVNoYXJlZElu
dGVycnVwdGlibHkoQWJzdHJhY3RRdWV1ZWRTeW5jaHJvbml6ZXIuamF2YToxMjgxKQogICAg
ICAgIGF0IGphdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2skU3luYy5pbm5lckdldChG
dXR1cmVUYXNrLmphdmE6MjE4KQogICAgICAgIGF0IGphdmEudXRpbC5jb25jdXJyZW50LkZ1
dHVyZVRhc2suZ2V0KEZ1dHVyZVRhc2suamF2YTo4MykKICAgICAgICBhdCBvcmcuYXBhY2hl
LmNhc3NhbmRyYS51dGlscy5GQlV0aWxpdGllcy53YWl0T25GdXR1cmUoRkJVdGlsaXRpZXMu
amF2YTozNzUpCiAgICAgICAgYXQgb3JnLmFwYWNoZS5jYXNzYW5kcmEudXRpbHMuRkJVdGls
aXRpZXMud2FpdE9uRnV0dXJlcyhGQlV0aWxpdGllcy5qYXZhOjM2OCkKICAgICAgICBhdCBv
cmcuYXBhY2hlLmNhc3NhbmRyYS5zdHJlYW1pbmcuU3RyZWFtT3V0LmZsdXNoU1NUYWJsZXMo
U3RyZWFtT3V0LmphdmE6MTA4KQogICAgICAgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLnN0
cmVhbWluZy5TdHJlYW1PdXQudHJhbnNmZXJSYW5nZXMoU3RyZWFtT3V0LmphdmE6MTM2KQog
ICAgICAgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLnN0cmVhbWluZy5TdHJlYW1PdXQudHJh
bnNmZXJSYW5nZXMoU3RyZWFtT3V0LmphdmE6MTE2KQogICAgICAgIGF0IG9yZy5hcGFjaGUu
Y2Fzc2FuZHJhLnN0cmVhbWluZy5TdHJlYW1SZXF1ZXN0VmVyYkhhbmRsZXIuZG9WZXJiKFN0
cmVhbVJlcXVlc3RWZXJiSGFuZGxlci5qYXZhOjQ0KQogICAgICAgIGF0IG9yZy5hcGFjaGUu
Y2Fzc2FuZHJhLm5ldC5NZXNzYWdlRGVsaXZlcnlUYXNrLnJ1bihNZXNzYWdlRGVsaXZlcnlU
YXNrLmphdmE6NTYpCiAgICAgICAgYXQgamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9v
bEV4ZWN1dG9yJFdvcmtlci5ydW5UYXNrKFRocmVhZFBvb2xFeGVjdXRvci5qYXZhOjg4NikK
ICAgICAgICBhdCBqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29y
a2VyLnJ1bihUaHJlYWRQb29sRXhlY3V0b3IuamF2YTo5MDgpCiAgICAgICAgYXQgamF2YS5s
YW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6NjYyKQoK
--------------000504070701010101040304--