Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 69096 invoked from network); 7 Feb 2011 23:03:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Feb 2011 23:03:55 -0000 Received: (qmail 51896 invoked by uid 500); 7 Feb 2011 23:03:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 51812 invoked by uid 500); 7 Feb 2011 23:03:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51803 invoked by uid 99); 7 Feb 2011 23:03:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 23:03:52 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 74.125.83.172 as permitted sender) Received: from [74.125.83.172] (HELO mail-pv0-f172.google.com) (74.125.83.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 23:03:45 +0000 Received: by pvc21 with SMTP id 21so1268413pvc.31 for ; Mon, 07 Feb 2011 15:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=J9AQrocvYdRQDpPyeTa8mQx/Bb+VUWojiiiRsNLq1ME=; b=Mh2cySI1BtUfM7H448chPMcVJzFmXU4f0UfpuSzRj1lLXECBmtpipl0uN0kkVQtV1a VC+A4b1YiGjc9hUCUsleAbK5jSpQRRqIFz66IgYEl2TE0tLtNQ9I5G8groiHsVsjnSzb 9N4cT7IOm+sMn0ugrBE7K27LyBAovy3WDVKN8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=l8K8SGJqoNE/kUr+G0PO7wKTIoUnesE3oX6jCcZReDKx1Z5XwYyr0fXMxRVaijLaZp guQFFExm9GnafNEcqqFV62zDRi/jVDnPXorgK+D9fqAlbpa3YRDNOv0FNdYemgyJzxV9 SYWeSBAJVgNcdbUk/qcV5U8kij391YSrdxXN4= MIME-Version: 1.0 Received: by 10.142.50.20 with SMTP id x20mr4899765wfx.288.1297119803972; Mon, 07 Feb 2011 15:03:23 -0800 (PST) Received: by 10.142.164.19 with HTTP; Mon, 7 Feb 2011 15:03:23 -0800 (PST) In-Reply-To: References: Date: Mon, 7 Feb 2011 17:03:23 -0600 Message-ID: Subject: Re: [0.7.1] Error in ThreadPoolExecutor From: Jonathan Ellis To: user Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Can you open a ticket for this? And are you using order-preserving partiti= oner? On Mon, Feb 7, 2011 at 7:16 AM, Patrik Modesto w= rote: > Hi, > > on my two-node test setup I get repeatedly following error: > > The 10.0.18.129 server log: > > =A0INFO 14:10:37,707 Node /10.0.18.99 has restarted, now UP again > =A0INFO 14:10:37,708 Checking remote schema before delivering hints > =A0INFO 14:10:37,708 Sleeping 45506ms to stagger hint delivery > =A0INFO 14:10:37,709 Node /10.0.18.99 state jump to normal > =A0INFO 14:11:23,215 Started hinted handoff for endpoint /10.0.18.99 > ERROR 14:11:23,884 Error in ThreadPoolExecutor > java.lang.RuntimeException: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:34) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at java.nio.Buffer.position(Buffer.java:218) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:117) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:111) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.getTableAn= dCFNames(HintedHandOffManager.java:237) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:306) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.access$100= (HintedHandOffManager.java:88) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager$2.runMayTh= row(HintedHandOffManager.java:385) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0... 3 more > ERROR 14:11:23,885 Fatal exception in thread Thread[HintedHandoff:1,1,mai= n] > java.lang.RuntimeException: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:34) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at java.nio.Buffer.position(Buffer.java:218) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:117) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:111) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.getTableAn= dCFNames(HintedHandOffManager.java:237) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:306) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.access$100= (HintedHandOffManager.java:88) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager$2.runMayTh= row(HintedHandOffManager.java:385) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0... 3 more > > > The 10.0.18.99 server log: > > =A0INFO 14:10:37,691 Binding thrift service to /0.0.0.0:9160 > =A0INFO 14:10:37,693 Using TFastFramedTransport with a max frame size of > 15728640 bytes. > =A0INFO 14:10:37,695 Listening for thrift clients... > =A0INFO 14:10:38,337 GC for ParNew: 954 ms, 658827608 reclaimed leaving > 966732432 used; max is 4265607168 > =A0INFO 14:11:27,142 Started hinted handoff for endpoint /10.0.18.129 > ERROR 14:11:27,370 Error in ThreadPoolExecutor > java.lang.RuntimeException: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:34) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at java.nio.Buffer.position(Buffer.java:218) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:117) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:111) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.getTableAn= dCFNames(HintedHandOffManager.java:237) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:306) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.access$100= (HintedHandOffManager.java:88) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager$2.runMayTh= row(HintedHandOffManager.java:385) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0... 3 more > ERROR 14:11:27,371 Fatal exception in thread Thread[HintedHandoff:1,1,mai= n] > java.lang.RuntimeException: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:34) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.IllegalArgumentException > =A0 =A0 =A0 =A0at java.nio.Buffer.position(Buffer.java:218) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:117) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBu= fferUtil.java:111) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.getTableAn= dCFNames(HintedHandOffManager.java:237) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:306) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.access$100= (HintedHandOffManager.java:88) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager$2.runMayTh= row(HintedHandOffManager.java:385) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0... 3 more > > It happen durring batch_mutate test or after restart, when there are > commitlogs to replay. Using current 0.7.1 from cassandra-0.7 branch. > > HTH, > Patrik > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com