From user-return-27533-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Jul 10 06:21:34 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71B11D65F for ; Tue, 10 Jul 2012 06:21:34 +0000 (UTC) Received: (qmail 49784 invoked by uid 500); 10 Jul 2012 06:21:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 49631 invoked by uid 500); 10 Jul 2012 06:21:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 49594 invoked by uid 99); 10 Jul 2012 06:21:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jul 2012 06:21:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [80.252.97.80] (HELO mailout.artfiles.de) (80.252.97.80) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jul 2012 06:21:23 +0000 Received: from [62.143.152.70] (helo=[192.168.0.169]) auth=info@overtronic.de by mailout.artfiles.de with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.72) id 1SoToh-0000Z7-RE for user@cassandra.apache.org; Tue, 10 Jul 2012 08:20:59 +0200 From: =?iso-8859-1?Q?Ivo_Mei=DFner?= Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_1AC74273-773A-4974-93C3-5BB56920FADB" Subject: Re: BulkLoading sstables from v1.0.3 to v1.1.1 Date: Tue, 10 Jul 2012 08:20:59 +0200 In-Reply-To: <517916DB-CEFB-4038-9098-10B3F0FF28C8@gmail.com> To: user@cassandra.apache.org References: <517916DB-CEFB-4038-9098-10B3F0FF28C8@gmail.com> Message-Id: <2B310EFC-A9D5-44F2-8AB2-7EBEA0C440FC@overtronic.com> X-Mailer: Apple Mail (2.1278) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_1AC74273-773A-4974-93C3-5BB56920FADB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, there are some problems in version 1.1.1 with secondary indexes and key = caches that are fixed in 1.1.2.=20 I would try to upgrade to 1.1.2 and see if the error still occurs.=20 Ivo >=20 >=20 > Hi=20 >=20 > As part of a continuous development of a system migration, we have a = test build to take a snapshot of a keyspace from cassandra v 1.0.3 and = bulk load it to a cluster of 1.1.1 using the sstableloader.sh. Not sure = if relevant, but one of the cf contains a secondary index.=20 >=20 > The build basically does:=20 > Drop the destination keyspace if exist=20 > Add the destination keyspace, wait for schema to agree=20 > run sstableLoader=20 > Do some validation of the streamed data=20 >=20 > Keyspace / column families schema are basically the same, apart from = in the one of v1.1.1, we had compression and key cache switched on.=20 >=20 > On a clean cluster, (empty data, commit log, saved-cache dirs) the = sstables loaded beautifully.=20 >=20 > But subsequent build failed with=20 > --=20 > [21:02:02][exec] progress: []... [total: 0 - 0MB/s = (avg: 0MB/s)]ERROR 21:02:02,811 Error in = ThreadPoolExecutorjava.lang.RuntimeException: java.net.SocketException: = Connection reset=20 > [21:02:02][exec] at = org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)=20= > [21:02:02][exec] at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)=20= > [21:02:02][exec] at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)=20 > [21:02:02][exec] at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908)=20 > [21:02:02][exec] at java.lang.Thread.run(Thread.java:662)=20 > [21:02:02][exec] Caused by: java.net.SocketException: Connection reset=20= > [21:02:02][exec] at = java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)=20 > [21:02:02][exec] at = java.net.SocketOutputStream.write(SocketOutputStream.java:136)=20 > [21:02:02][exec] at = com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:1= 33)=20 > [21:02:02][exec] at = com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream= .java:203)=20 > [21:02:02][exec] at = com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97)=20 > [21:02:02][exec] at = org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:22= 7)=20 > [21:02:02][exec] at = org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:1= 68)=20 > [21:02:02][exec] at = org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.j= ava:94)=20 > [21:02:02][exec] at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)=20= > [21:02:02][exec] ... 3 more=20 > --=20 >=20 > If I look at the server log at this time, we have=20 > ---=20 > ERROR [Thread-30] 2012-07-07 21:02:44,484 AbstractCassandraDaemon.java = (line 134) Exception in thread Thread[Thread-30,5,main]=20 > java.lang.RuntimeException: java.util.concurrent.ExecutionException: = java.lang.AssertionError: = DecoratedKey(106448592537980973961347479329396275945, = 6e6669677c323031322d30362 at = org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryInd= exes(SecondaryIndexManager.java:136)=20 > at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:202)=20 > at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:103)=20 > at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:182)=20 > at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:78)=20 > --=20 >=20 > =46rom then on, we had different errors with the rest of the bulkLoad:=20= > --=20 > ERROR [Thread-54] 2012-07-07 21:04:33,589 AbstractCassandraDaemon.java = (line 134) Exception in thread Thread[Thread-54,5,main]=20 > java.lang.AssertionError: We shouldn't fail acquiring a reference on a = sstable that has just been transferred=20 > at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:188)=20 > at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:103)=20 > at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:182)=20 > at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:78)=20 > --=20 >=20 > We would like to understand this error. It's our believe that you = could reload the same set of sstables without problem.=20 >=20 > Have we done anything wrong? Many thanks for your help.=20 >=20 > - A=20 >=20 --Apple-Mail=_1AC74273-773A-4974-93C3-5BB56920FADB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii


Hi 

As part of a continuous = development of a system migration, we have a test build to take a = snapshot of a keyspace from cassandra v 1.0.3 and bulk load it = to a cluster of 1.1.1 using the sstableloader.sh.  Not sure if = relevant, but one of the cf contains a secondary index. 

The build basically = does: 
Drop the destination keyspace if = exist 
Add the destination keyspace, wait for = schema to agree 
run = sstableLoader 
Do some validation of the = streamed data 
Keyspace / column families schema are = basically the same, apart from in the one of v1.1.1, we had compression = and key cache switched on. 

On a clean cluster, (empty data, = commit log, saved-cache dirs) the sstables loaded = beautifully. 
But subsequent build failed = with 
-- 
[21:02:02][exec] progress: = [<snip ip_addresses>]... [total: 0 - 0MB/s (avg: 0MB/s)]ERROR = 21:02:02,811 Error in ThreadPoolExecutorjava.lang.RuntimeException: = java.net.SocketException: Connection reset 
[21:02:02][exec] at = org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)&nbs= p;
[21:02:02][exec] at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)&nb= sp;
[21:02:02][exec] at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886) 
[21:02:02][exec] at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908) 
[21:02:02][exec] at = java.lang.Thread.run(Thread.java:662) 
[21:02:02][exec] Caused by: = java.net.SocketException: Connection reset 
[21:02:02][exec] at = java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) <= /span>
[21:02:02][exec] at = java.net.SocketOutputStream.write(SocketOutputStream.java:136) 
[21:02:02][exec] at = com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:1= 33) 
[21:02:02][exec] at = com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream= .java:203) 

[21:02:02][exec] at = com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97) =

[21:02:02][exec] at = org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:22= 7) 
[21:02:02][exec] at = org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:1= 68) 
[21:02:02][exec] at = org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.j= ava:94) 
[21:02:02][exec] at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)&nb= sp;
[21:02:02][exec] ... 3 = more 
-- 

If I look at the server log at = this time, we have 
--- 
ERROR [Thread-30] 2012-07-07 = 21:02:44,484 AbstractCassandraDaemon.java (line 134) Exception in thread = Thread[Thread-30,5,main] 
java.lang.RuntimeException: = java.util.concurrent.ExecutionException: java.lang.AssertionError: = DecoratedKey(106448592537980973961347479329396275945, = 6e6669677c323031322d30362<snip..., v long key)) !=3D = DecoratedKey(155376897532138582317079439091276375324, = 444956334f6666696369616c2d5969656c644375727665737c323031322d30352d3331) = in = /opt/cassandra/data/dev_load_test2/journal/dev_load_test2-journal-hd-5-Dat= a.db 
        at = org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryInd= exes(SecondaryIndexManager.java:136) 
        at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:202) 
        at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:103) 
        at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:182) 
        at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:78) 
-- 

=46rom then on, we had different = errors with the rest of the bulkLoad: 
-- 
ERROR [Thread-54] 2012-07-07 = 21:04:33,589 AbstractCassandraDaemon.java (line 134) Exception in thread = Thread[Thread-54,5,main] 
java.lang.AssertionError: We = shouldn't fail acquiring a reference on a sstable that has just been = transferred 
        at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:188) 
        at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:103) 
        at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:182) 
        at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:78) 
-- 

We would like to understand this = error.  It's our believe that you could reload the same set of = sstables without problem. 
Have we done anything wrong? =  Many thanks for your help. 

- = A 



= --Apple-Mail=_1AC74273-773A-4974-93C3-5BB56920FADB--