cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kjellman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-5229) after a IOException is thrown during streaming, streaming tasks hang in netstats
Date Thu, 07 Feb 2013 20:45:13 GMT
Michael Kjellman created CASSANDRA-5229:
-------------------------------------------

             Summary: after a IOException is thrown during streaming, streaming tasks hang
in netstats
                 Key: CASSANDRA-5229
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5229
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.2.1
         Environment: Ubuntu 12.04
            Reporter: Michael Kjellman
            Priority: Critical


After an IOExcpetion, streaming tasks marked as "successful" in the logs are hung in netstats

With TRACE debugging on streaming on the receiving node everything about the sstable in the
log (not very much)

{code}
 INFO [AntiEntropyStage:1] 2013-02-07 11:23:44,717 StreamOut.java (line 151) Stream context
metadata [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db sections=3068
progress=0/2785204713 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db
sections=2696 progress=0/758409465 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db
sections=3099 progress=0/238876436 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db
sections=1166 progress=0/2125323 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db
sections=2507 progress=0/515992757 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
sections=3153 progress=0/994857654 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db
sections=3116 progress=0/129398170 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db
sections=217 progress=0/72286 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db
sections=3146 progress=0/3357709019 - 0%], 27 sstables.
 INFO [AntiEntropyStage:1] 2013-02-07 11:23:52,964 StreamOut.java (line 151) Stream context
metadata [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db sections=2930
progress=0/2799914560 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db
sections=2590 progress=0/761266059 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db
sections=2956 progress=0/241362497 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db
sections=1153 progress=0/2125323 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db
sections=2422 progress=0/522126371 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
sections=3004 progress=0/998401202 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db
sections=2974 progress=0/129722346 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db
sections=220 progress=0/72286 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db
sections=2998 progress=0/3375554099 - 0%], 27 sstables.
{code}

node that is streaming out thinks that the streaming session was successful 
{code}
 INFO [MiscStage:1] 2013-02-07 11:23:38,022 StreamOut.java (line 151) Stream context metadata
[/data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db sections=1727
progress=0/210208515 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
sections=1746 progress=0/119438030 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db
sections=1681 progress=0/54498226 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db
sections=16 progress=0/13490 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db
sections=632 progress=0/70019542 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db
sections=1644 progress=0/39870238 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db
sections=1569 progress=0/208331077 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db
sections=1572 progress=0/30870478 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db
sections=167 progress=0/1845123 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db
sections=1574 progress=0/287386471 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db
sections=811 progress=0/103776521 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db
sections=1539 progress=0/141864261 - 0%], 12 sstables.
 INFO [MiscStage:1] 2013-02-07 11:23:49,938 StreamOut.java (line 151) Stream context metadata
[/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db sections=3153
progress=0/994857654 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db
sections=2507 progress=0/515992757 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db
sections=3153 progress=0/131969347 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db
sections=3068 progress=0/2784807967 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db
sections=2696 progress=0/758409465 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db
sections=3150 progress=0/1792868996 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db
sections=3123 progress=0/240094510 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-929-Data.db
sections=18 progress=0/29468 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db
sections=217 progress=0/13490 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db
sections=3124 progress=0/130320291 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db
sections=3055 progress=0/1749539598 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db
sections=1608 progress=0/13357410 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db
sections=3096 progress=0/1241397203 - 0%], 13 sstables.
 INFO [Streaming to /10.8.25.132:2] 2013-02-07 11:24:18,780 StreamReplyVerbHandler.java (line
44) Successfully sent /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
to /10.8.25.132
{code}

node that is being streamed to has nothing in the logs about this particular sstable


{code}
Streaming from: /10.8.30.13
evidence: /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
sections=3153 progress=218225400/994857654 - 21%

_______

Streaming from: /10.138.12.10
evidence: /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db sections=3153
progress=218225400/994857654 - 21%
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message