Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 881E3E61F for ; Thu, 14 Feb 2013 05:58:15 +0000 (UTC) Received: (qmail 96936 invoked by uid 500); 14 Feb 2013 05:58:14 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 96758 invoked by uid 500); 14 Feb 2013 05:58:14 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 96689 invoked by uid 99); 14 Feb 2013 05:58:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Feb 2013 05:58:13 +0000 Date: Thu, 14 Feb 2013 05:58:13 +0000 (UTC) From: "Jonathan Ellis (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-5229) after a IOException is thrown during streaming, streaming tasks hang in netstats MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5229?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-5229: -------------------------------------- Priority: Minor (was: Critical) =20 > after a IOException is thrown during streaming, streaming tasks hang in n= etstats > -------------------------------------------------------------------------= ------- > > Key: CASSANDRA-5229 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5229 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.2.1 > Environment: Ubuntu 12.04 > Reporter: Michael Kjellman > Assignee: Yuki Morishita > Priority: Minor > Fix For: 1.2.2 > > > After an IOExcpetion, streaming tasks marked as "successful" in the logs = are hung in netstats > With TRACE debugging on streaming on the receiving node everything about = the sstable in the log (not very much) > {code} > INFO [AntiEntropyStage:1] 2013-02-07 11:23:44,717 StreamOut.java (line 1= 51) Stream context metadata [/data/cassandra/evidence/fingerprints/evidence= -fingerprints-ib-5-Data.db sections=3D3068 progress=3D0/2785204713 - 0%, /d= ata2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db se= ctions=3D2696 progress=3D0/758409465 - 0%, /data/cassandra/evidence/fingerp= rints/evidence-fingerprints-ib-60-Data.db sections=3D3099 progress=3D0/2388= 76436 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib= -63-Data.db sections=3D1166 progress=3D0/2125323 - 0%, /data2/cassandra/evi= dence/fingerprints/evidence-fingerprints-ib-38-Data.db sections=3D2507 prog= ress=3D0/515992757 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fi= ngerprints-ib-26-Data.db sections=3D3153 progress=3D0/994857654 - 0%, /data= 2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db secti= ons=3D3116 progress=3D0/129398170 - 0%, /data2/cassandra/evidence/fingerpri= nts/evidence-fingerprints-ib-58-Data.db sections=3D217 progress=3D0/72286 -= 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data= .db sections=3D3146 progress=3D0/3357709019 - 0%], 27 sstables. > INFO [AntiEntropyStage:1] 2013-02-07 11:23:52,964 StreamOut.java (line 1= 51) Stream context metadata [/data/cassandra/evidence/fingerprints/evidence= -fingerprints-ib-5-Data.db sections=3D2930 progress=3D0/2799914560 - 0%, /d= ata2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db se= ctions=3D2590 progress=3D0/761266059 - 0%, /data/cassandra/evidence/fingerp= rints/evidence-fingerprints-ib-60-Data.db sections=3D2956 progress=3D0/2413= 62497 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib= -63-Data.db sections=3D1153 progress=3D0/2125323 - 0%, /data2/cassandra/evi= dence/fingerprints/evidence-fingerprints-ib-38-Data.db sections=3D2422 prog= ress=3D0/522126371 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fi= ngerprints-ib-26-Data.db sections=3D3004 progress=3D0/998401202 - 0%, /data= 2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db secti= ons=3D2974 progress=3D0/129722346 - 0%, /data2/cassandra/evidence/fingerpri= nts/evidence-fingerprints-ib-58-Data.db sections=3D220 progress=3D0/72286 -= 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data= .db sections=3D2998 progress=3D0/3375554099 - 0%], 27 sstables. > {code} > node that is streaming out thinks that the streaming session was successf= ul=20 > {code} > INFO [MiscStage:1] 2013-02-07 11:23:38,022 StreamOut.java (line 151) Str= eam context metadata [/data/cassandra/evidence/fingerprints/evidence-finger= prints-ia-472-Data.db sections=3D1727 progress=3D0/210208515 - 0%, /var/lib= /cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db= sections=3D1746 progress=3D0/119438030 - 0%, /data/cassandra/evidence/fing= erprints/evidence-fingerprints-ib-920-Data.db sections=3D1681 progress=3D0/= 54498226 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fing= erprints-ib-922-Data.db sections=3D16 progress=3D0/13490 - 0%, /var/lib/cas= sandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db sec= tions=3D632 progress=3D0/70019542 - 0%, /var/lib/cassandra/data2/evidence/f= ingerprints/evidence-fingerprints-ib-921-Data.db sections=3D1644 progress= =3D0/39870238 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerpr= ints-ib-497-Data.db sections=3D1569 progress=3D0/208331077 - 0%, /var/lib/c= assandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db s= ections=3D1572 progress=3D0/30870478 - 0%, /var/lib/cassandra/data2/evidenc= e/fingerprints/evidence-fingerprints-ib-925-Data.db sections=3D167 progress= =3D0/1845123 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-= fingerprints-ib-703-Data.db sections=3D1574 progress=3D0/287386471 - 0%, /v= ar/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-D= ata.db sections=3D811 progress=3D0/103776521 - 0%, /var/lib/cassandra/data2= /evidence/fingerprints/evidence-fingerprints-ib-915-Data.db sections=3D1539= progress=3D0/141864261 - 0%], 12 sstables. > INFO [MiscStage:1] 2013-02-07 11:23:49,938 StreamOut.java (line 151) Str= eam context metadata [/var/lib/cassandra/data2/evidence/fingerprints/eviden= ce-fingerprints-ib-919-Data.db sections=3D3153 progress=3D0/994857654 - 0%,= /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-91= 8-Data.db sections=3D2507 progress=3D0/515992757 - 0%, /var/lib/cassandra/d= ata2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db sections=3D= 3153 progress=3D0/131969347 - 0%, /var/lib/cassandra/data2/evidence/fingerp= rints/evidence-fingerprints-ib-703-Data.db sections=3D3068 progress=3D0/278= 4807967 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-finge= rprints-ib-913-Data.db sections=3D2696 progress=3D0/758409465 - 0%, /data/c= assandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db section= s=3D3150 progress=3D0/1792868996 - 0%, /data/cassandra/evidence/fingerprint= s/evidence-fingerprints-ib-920-Data.db sections=3D3123 progress=3D0/2400945= 10 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprin= ts-ib-929-Data.db sections=3D18 progress=3D0/29468 - 0%, /var/lib/cassandra= /data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db sections= =3D217 progress=3D0/13490 - 0%, /var/lib/cassandra/data2/evidence/fingerpri= nts/evidence-fingerprints-ib-921-Data.db sections=3D3124 progress=3D0/13032= 0291 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-4= 97-Data.db sections=3D3055 progress=3D0/1749539598 - 0%, /var/lib/cassandra= /data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db sections= =3D1608 progress=3D0/13357410 - 0%, /var/lib/cassandra/data2/evidence/finge= rprints/evidence-fingerprints-ib-915-Data.db sections=3D3096 progress=3D0/1= 241397203 - 0%], 13 sstables. > INFO [Streaming to /10.8.25.132:2] 2013-02-07 11:24:18,780 StreamReplyVe= rbHandler.java (line 44) Successfully sent /var/lib/cassandra/data2/evidenc= e/fingerprints/evidence-fingerprints-ib-919-Data.db to /10.8.25.132 > {code} > node that is being streamed to has nothing in the logs about this particu= lar sstable > {code} > Streaming from: /10.8.30.13 > evidence: /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerp= rints-ib-919-Data.db sections=3D3153 progress=3D218225400/994857654 - 21% > _______ > Streaming from: /10.138.12.10 > evidence: /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib= -26-Data.db sections=3D3153 progress=3D218225400/994857654 - 21% > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira