From user-return-35397-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Jul 18 17:17:49 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B3F5BC73A for ; Thu, 18 Jul 2013 17:17:49 +0000 (UTC) Received: (qmail 77653 invoked by uid 500); 18 Jul 2013 17:17:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 76891 invoked by uid 500); 18 Jul 2013 17:17:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 76365 invoked by uid 99); 18 Jul 2013 17:17:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jul 2013 17:17:42 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [88.198.227.104] (HELO team.enercast.de) (88.198.227.104) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jul 2013 17:17:37 +0000 Received: from [100.95.140.200] (tmo-096-80.customers.d1-online.com [80.187.96.80]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by team.enercast.de (Postfix) with ESMTPSA id 10451B400EA; Thu, 18 Jul 2013 17:17:13 +0000 (UTC) References: <51E80F2A.6030302@enercast.de> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: 7bit Content-Type: multipart/alternative; boundary=Apple-Mail-E6F61E11-7D07-40FE-9BE5-1B071331A31E Message-Id: Cc: "user@cassandra.apache.org" X-Mailer: iPhone Mail (10B350) From: Jan Kesten Subject: Re: Corrupted sstable and sstableloader Date: Thu, 18 Jul 2013 19:17:10 +0200 To: "user@cassandra.apache.org" X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-E6F61E11-7D07-40FE-9BE5-1B071331A31E Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, I think it might be corrupted due to a poweroutage. Apart from this issue re= ading the data with consistency level quorum (I have three replicas) did not= issue an error - only the import to a different cluster. So, if I import all nodes except the one with the corrupted sstable - shoudn= 't I import two of the three replicas, so that the data is complete? Von meinem iPhone gesendet Am 18.07.2013 um 19:06 schrieb sankalp kohli : > sstable might be corrupted due to bad disk. In that case, replication does= not matter. >=20 >=20 > On Thu, Jul 18, 2013 at 8:52 AM, Jan Kesten wrote: >> Hello together, >>=20 >> today I experienced a problem while loading a snapshot from our cassandra= cluster to test cluster. The cluster has six nodes and I took a snapshot fr= om all nodes concurrently and tried to import them in the other cluster. >>=20 >> =46rom 5 out of 6 nodes importing went well with no errors. But one snaps= hot of one node cannot be imported - I tried serveral times. I got the follo= wing while running sstableloader: >>=20 >> ERROR 09:13:06,084 Error in ThreadPoolExecutor >> java.lang.RuntimeException: java.io.IOException: Daten=C3=BCbergabe unter= brochen (broken pipe) >> at com.google.common.base.Throwables.propagate(Throwables.java:160) >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.jav= a:32) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:615) >> at java.lang.Thread.run(Thread.java:724) >> Caused by: java.io.IOException: Daten=C3=BCbergabe unterbrochen (broken p= ipe) >> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) >> at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java= :420) >> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:552) >> at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.s= tream(CompressedFileStreamTask.java:93) >> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStre= amTask.java:91) >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.jav= a:28) >> ... 3 more >> Exception in thread "Streaming to /172.17.2.216:1" java.lang.RuntimeExcep= tion: java.io.IOException: Daten=C3=BCbergabe unterbrochen (broken pipe) >> at com.google.common.base.Throwables.propagate(Throwables.java:160) >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.jav= a:32) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:615) >> at java.lang.Thread.run(Thread.java:724) >> Caused by: java.io.IOException: Daten=C3=BCbergabe unterbrochen (broken p= ipe) >> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) >> at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java= :420) >> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:552) >> at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.s= tream(CompressedFileStreamTask.java:93) >> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStre= amTask.java:91) >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.jav= a:28) >> ... 3 more >>=20 >> I suspect that the sstable on the node is corrupted in some way - and a s= crub and repair should fix that I suppose. >>=20 >> Since the original cluster has a replication factor of 3 - shoudn't the i= mport from 5 of 6 snapshots contain all data? Or is the sstableloader tool t= oo clever and avoids importing double data? >>=20 >> Thanks for hints, >> Jan >>=20 >> --=20 >> Jan Kesten, mailto:j.kesten@enercast.de >> Tel.: +49 561/4739664-0 FAX: -9 >> enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 >> http://www.enercast.de Online-Prognosen f=C3=BCr erneuerbare Energien >> Gesch=C3=A4ftsf=C3=BChrung: Dipl. Ing. Thomas Landgraf, Bernd Kratz >>=20 >> Diese E-Mail und etwaige Anh=C3=A4nge k=C3=B6nnen vertrauliche und/oder r= echtlich gesch=C3=BCtzte Informationen enthalten. Falls Sie nicht der angege= bene Empf=C3=A4nger sind oder falls diese E-Mail irrt=C3=BCmlich an Sie adre= ssiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und l= =C3=B6schen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso= d=C3=BCrfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritt= e weitergeben. Vielen Dank. >>=20 >> This e-mail and any attachment may contain confidential and/or privileged= information. If you are not the named addressee or if this transmission has= been addressed to you in error, please notify us immediately by reply e-mai= l and then delete this e-mail and any attachment from your system. Please un= derstand that you must not copy this e-mail or any attachment or disclose th= e contents to any other person. Thank you for your cooperation. >=20 --Apple-Mail-E6F61E11-7D07-40FE-9BE5-1B071331A31E Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hi,

I think i= t might be corrupted due to a poweroutage. Apart from this issue reading the= data with consistency level quorum (I have three replicas) did not issue an= error - only the import to a different cluster.

So= , if I import all nodes except the one with the corrupted sstable - shoudn't= I import two of the three replicas, so that the data is complete?


Von meinem iPhone gesendet

Am 18.07.2013 um 19:06 sch= rieb sankalp kohli <kohlisankal= p@gmail.com>:

sstable might be corrupted due to bad disk. In that case, replication d= oes not matter.


On Thu, Jul 18, 2013 at 8:52 AM, Jan Kesten <j.kesten@enercast.de> wrote:
Hello together,

today I experienced a problem while loading a snapshot from our cassandra cl= uster to test cluster. The cluster has six nodes and I took a snapshot from a= ll nodes concurrently and tried to import them in the other cluster.

=46rom 5 out of 6 nodes importing went well with no errors. But one snapshot= of one node cannot be imported - I tried serveral times. I got the followin= g while running sstableloader:

ERROR 09:13:06,084 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.IOException: Daten=C3=BCbergabe unterbro= chen (broken pipe)
    at com.google.common.base.Throwables.propagate(T= hrowables.java:160)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(<= u>ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Daten=C3=BCbergabe unterbrochen (broken pipe= )
    at sun.nio.ch.FileChannelImpl.transferTo0(Native Method= )
    at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:420)
    at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImp= l.java:552)
    at org.apache.cassandra.streaming.compress.Compr= essedFileStreamTask.stream(CompressedFileStreamTask.java:93)
    at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    ... 3 more
Exception in thread "Streaming to /
172.17.2.216:1" java.lang.RuntimeException: java.io.IOException= : Daten=C3=BCbergabe unterbrochen (broken pipe)
    at com.google.common.base.Throwables.propagate(T= hrowables.java:160)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(<= u>
ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Daten=C3=BCbergabe unterbrochen (broken pipe= )
    at sun.nio.ch.FileChannelImpl.transferTo0(Native Method= )
    at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:420)
    at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImp= l.java:552)
    at org.apache.cassandra.streaming.compress.Compr= essedFileStreamTask.stream(CompressedFileStreamTask.java:93)
    at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    ... 3 more

I suspect that the sstable on the node is corrupted in some way - and a scru= b and repair should fix that I suppose.

Since the original cluster has a replication factor of 3 - shoudn't the impo= rt from 5 of 6 snapshots contain all data? Or is the sstableloader tool too c= lever and avoids importing double data?

Thanks for hints,
Jan

--
Jan Kesten, mailto:j.kesten@enercast.de
Tel.: +49 561/4739664-0 FAX: -9
enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel       H= RB15471
http://www.enercast.de<= /a> Online-Prognosen f=C3=BCr erneuerbare Energien
Gesch=C3=A4ftsf=C3=BChrung: Dipl. Ing. Thomas Landgraf, Bernd Kratz

Diese E-Mail und etwaige Anh=C3=A4nge k=C3=B6nnen vertrauliche und/oder rech= tlich gesch=C3=BCtzte Informationen enthalten. Falls Sie nicht der angegeben= e Empf=C3=A4nger sind oder falls diese E-Mail irrt=C3=BCmlich an Sie adressi= ert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und l=C3= =B6schen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso d=C3= =BCrfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte wei= tergeben. Vielen Dank.

This e-mail and any attachment may contain confidential and/or privileged in= formation. If you are not the named addressee or if this transmission has be= en addressed to you in error, please notify us immediately by reply e-mail a= nd then delete this e-mail and any attachment from your system. Please under= stand that you must not copy this e-mail or any attachment or disclose the c= ontents to any other person. Thank you for your cooperation.


= --Apple-Mail-E6F61E11-7D07-40FE-9BE5-1B071331A31E--