Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0CDC10AD3 for ; Sun, 8 Sep 2013 22:23:15 +0000 (UTC) Received: (qmail 23798 invoked by uid 500); 8 Sep 2013 22:23:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23735 invoked by uid 500); 8 Sep 2013 22:23:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23727 invoked by uid 99); 8 Sep 2013 22:23:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Sep 2013 22:23:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [209.85.215.47] (HELO mail-la0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Sep 2013 22:23:07 +0000 Received: by mail-la0-f47.google.com with SMTP id eo20so4330786lab.34 for ; Sun, 08 Sep 2013 15:22:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=cX9uwwlbRxqQXR/SuM6gCSC1RRNOu8seT2IqaYs5Hdo=; b=lLw6/I2OyXS/Q4KcuSb/XmBCxF0eowIv6a8VtL2ddlhR8f1y4HB3pfH3h+IQpVANsz mE85bg38cci7hxz4At+Z5HJg5RzeZgC+KRYoHhmvucP4I6VJGEfOjcx+V4HkQbvY2XNZ XgPEVVO/FGIwQwgip8Rg7fLwTb9e4JH6upCo18VTf5VKdIkHX2K6HJDDUJtakxFMC9C7 mTNrX0zbyUgtn3O5oajBhEjaDhHS1+ykCBYEUDIY+RUyxxd2iS79EkOYZfJCUUXvlxJE Vkvdwk6nslKwi3STIbmdlmzgWPeTAmqoCKQ6BW3ZdK/F0z0ou0vguRxZHkWA6GAjgsql fRaQ== X-Gm-Message-State: ALoCoQleWONBMD9ugPYX4nzas1CkDMcrW5Nk1B+jdTV/TcxCtvX+ggdqqfTDts6St6cyzltOMrtD MIME-Version: 1.0 X-Received: by 10.112.14.3 with SMTP id l3mr59743lbc.27.1378678945256; Sun, 08 Sep 2013 15:22:25 -0700 (PDT) Received: by 10.112.159.166 with HTTP; Sun, 8 Sep 2013 15:22:25 -0700 (PDT) X-Originating-IP: [96.246.196.139] Date: Sun, 8 Sep 2013 18:22:25 -0400 Message-ID: Subject: Help: Data Migration Errors From: Ben Waine To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c37a08caa52704e5e6b46b X-Virus-Checked: Checked by ClamAV on apache.org --001a11c37a08caa52704e5e6b46b Content-Type: text/plain; charset=ISO-8859-1 Hi - Can anyone help me with some Cas data migration issues I'm having? I'm attempting to migrate a dev ring (5 nodes) to a larger production one (6 nodes). Both are hosted on EC2. Cluster Info: Small: Cas v.1.2.6, Rep Factor 1, vnodes enabled Larger: Cas v.1.2.9, Rep Factor 3, vnodes enabled After a lot of reading round I decided to use sstableloader: http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra http://www.datastax.com/dev/blog/bulk-loading http://geekswithblogs.net/johnsPerfBlog/archive/2011/07/26/how-to-use-cassandrs-sstableloader.aspx I have a script which exports the target SS tables from each of the nodes in the smaller ring and moves them to an instance which can connect to the larger ring and run sstableloader. The script does the following on each node nodetool flush nodetool compact nodetool scrub scp to node specific directory on the target I've set up the sstable loader machine with Cas 1.2.9 and configured it to bind to it's static IP (listen_address, broadcast_address, rpc_address) and added a seed node from my target ring into the seed config. The security groups of the loader and the ring are both open to each other (all ports). Between tests I'm deleting the contnet of /mnt/cassandra/* and then creating a fresh schema (matching the smaller rings schema). I'm getting some errors when attempting to load data from the sstableloader instance. On the sstableloader instance: ERROR 22:03:30,409 Error in ThreadPoolExecutor java.lang.RuntimeException: java.io.IOException: Connection reset by peer at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.InputStream.read(InputStream.java:101) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81) at java.io.DataInputStream.readInt(DataInputStream.java:388) at org.apache.cassandra.net.MessageIn.read(MessageIn.java:60) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:197) at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:180) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Exception in thread "Streaming to /10.xxx.xxx.161:1" java.lang.RuntimeException: java.io.IOException: Connection reset by peer at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.InputStream.read(InputStream.java:101) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81) at java.io.DataInputStream.readInt(DataInputStream.java:388) at org.apache.cassandra.net.MessageIn.read(MessageIn.java:60) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:197) at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:180) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more And then the progress bar gets stuck like this: progress: [/10..xxx.xxx.199 1/1 (100)] [/10.xxx.xxx.75 1/1 (100)] [/10.xxx.xxx.228 1/1 (100)] [/10.xxx.xxx.243 1/1 (100)] [/10.xxx.xxx.46 1/1 (100)] [/10.xxx.xxx.161 0/1 (300)] [total: 149 - 0MB/s (avg: 0MB/s)] And on the instances in the ring: (node which accepts the request) DEBUG [Thrift:5] 2013-09-08 22:03:30,027 CustomTThreadPoolServer.java (line 209) Thrift transport error occurred during processing of message. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) (another node in the cluster) IOException reading from socket; closing java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:352) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:108) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:250) at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:185) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238) at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78) I'm pretty stuck, the process is pretty opaque to me. If anyone could help me out I'd really appreciate it! --001a11c37a08caa52704e5e6b46b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi -=A0
Can anyo= ne help me with some Cas data migration issues I'm having?=A0

I'm attempting to = migrate a dev ring (5 nodes) to a larger production one (6 nodes). Both are= hosted on EC2.=A0

Cluster Info:=A0
=

Small: Cas v.1.2.6, Rep Factor 1, vnodes enabled
Larger: Cas v.1.2.9, Rep Factor 3, vn= odes enabled

After = a lot of reading round I decided to use sstableloader:




I have a script which = exports the target SS tables from each of the nodes in the smaller ring and= moves them to an instance which can connect to the larger ring and run sst= ableloader. The script does the following on each node

nodetool flush
nodetool compact
nodetool scrub
scp to node = specific directory on the target

I'= ve set up the sstable loader machine with Cas 1.2.9 and configured it to bi= nd to it's static IP (listen_address, broadcast_address, rpc_address) a= nd added a seed node from my target ring into the seed config. The security= groups of the loader and the ring are both open to each other (all ports).= =A0

Between tests I'm = deleting the contnet of /mnt/cassandra/* and then creating a fresh schema (= matching the smaller rings schema).

I'm getting some e= rrors when attempting to load data from the sstableloader instance.

On the= sstableloader instance:=A0

ERROR 22:03:30,409 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.IOE= xception: Connection reset by peer
at com.google.common.base.Thro= wables.propagate(Throwables.java:160)
at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.run= Worker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.Thread= PoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.= io.IOException: Connection reset by peer
= at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDisp= atcher.java:39)
at sun.nio.ch.IOUtil.readIntoN= ativeBuffer(IOUtil.java:225)
at sun.nio.c= h.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.SocketChannelImpl.read(SocketCha= nnelImpl.java:375)
at sun.nio.ch.SocketAdaptor$So= cketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.InputStream.read(InputStream.java:1= 01)
at sun.nio.ch.ChannelInputStream.read= (ChannelInputStream.java:81)
at java.io.DataInputStream.readInt(DataInputSt= ream.java:388)
at org.apache.cassandra.net.Me= ssageIn.read(MessageIn.java:60)
at org.ap= ache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:19= 7)
at org.apache.cassandra.streaming.FileStreamTa= sk.stream(FileStreamTask.java:180)
at org.apache.cassandra.stream= ing.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunn= able.java:28)
... 3 more
Exception in thread "Streaming to /10.xxx= .xxx.161:1" java.lang.RuntimeException: java.io.IOException: Connectio= n reset by peer
at com.google.common.base.Throwables.propagate= (Throwables.java:160)
at org.apache.cassandra.utils.= WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1145)
at java.util.concurrent.ThreadPoolExecutor$Wor= ker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread= .java:724)
= Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherIm= pl.read0(Native Method)
at sun.nio.ch.Soc= ketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUt= il.java:225)
at sun.nio.ch.IOUtil.read(IOUt= il.java:198)
at sun.nio.ch.SocketChannelI= mpl.read(SocketChannelImpl.java:375)
at sun.nio.ch.SocketAdaptor$SocketInputStream.= read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStre= am.read(ChannelInputStream.java:103)
at j= ava.io.InputStream.read(InputStream.java:101)
at sun.nio.ch.ChannelInputStream.read(ChannelI= nputStream.java:81)
at java.io.DataInputStream.rea= dInt(DataInputStream.java:388)
at org.apa= che.cassandra.net.MessageIn.read(MessageIn.java:60)
at org.apache.cassandra.streaming.FileStreamTa= sk.receiveReply(FileStreamTask.java:197)
at org.apache.cassandra.stream= ing.FileStreamTask.stream(FileStreamTask.java:180)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(File= StreamTask.java:91)
at org.apache.cassandra.utils.WrappedRunnable.= run(WrappedRunnable.java:28)
... 3 more

And then the progress bar gets stuck= like this:=A0

progress: [/10..xxx.xx= x.199 1/1 (100)] [/10.xxx.xxx.75 1/1 (100)] [/10.xxx.xxx.228 1/1 (100)] [/1= 0.xxx.xxx.243 1/1 (100)] [/10.xxx.xxx.46 1/1 (100)] [/10.xxx.xxx.161 0/1 (3= 00)] [total: 149 - 0MB/s (avg: 0MB/s)]


And on the instances in th= e ring:=A0

(node which accepts th= e request)
=
DEBUG [Thr= ift:5] 2013-09-08 22:03:30,027 CustomTThreadPoolServer.java (line 209) Thri= ft transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
= at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTran= sport.java:132)
at org.apache.thrift.transport.TTransport.read= All(TTransport.java:84)
at org.apache.thrift.transport= .TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedT= ransport.java:101)
at org.apache.thrift.transport.TTransport.read= All(TTransport.java:84)
at org.apache.thrift.protocol.= TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProto= col.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.= readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TBaseProc= essor.process(TBaseProcessor.java:22)
at = org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(Custo= mTThreadPoolServer.java:199)
at java.util.concurrent.ThreadPoolExecutor.run= Worker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.Thread= PoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

(another node in the c= luster)
IOException read= ing from socket; closing
java.io.IOException: Corrupt (negative) value length encounte= red
at org.apache.cassandra.utils.ByteBufferUtil.r= eadWithLength(ByteBufferUtil.java:352)
at org.apache.cassandra.db.Col= umnSerializer.deserializeColumnBody(ColumnSerializer.java:108)
at org.apache.cassandra.db.OnDiskAtom$Serializer.dese= rializeFromSSTable(OnDiskAtom.java:92)
at org.apache.cassandra.io.sstable.SSTableWrit= er.appendFromStream(SSTableWriter.java:250)
at org.apache.cassandra.stream= ing.IncomingStreamReader.streamIn(IncomingStreamReader.java:185)
at org.apache.cassandra.streaming.IncomingStreamRea= der.read(IncomingStreamReader.java:122)
at org.apache.cassandra.net.IncomingTcpConnect= ion.stream(IncomingTcpConnection.java:238)
at org.apache.cassandra.net.In= comingTcpConnection.handleStream(IncomingTcpConnection.java:178)
at org.apache.cassandra.net.IncomingTcpConnection.r= un(IncomingTcpConnection.java:78)


I'm pretty stuck, the = process is pretty opaque to me. If anyone could help me out I'd really = appreciate it!
--001a11c37a08caa52704e5e6b46b--