Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 26F0079F1 for ; Tue, 9 Aug 2011 21:24:48 +0000 (UTC) Received: (qmail 7355 invoked by uid 500); 9 Aug 2011 21:24:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 7290 invoked by uid 500); 9 Aug 2011 21:24:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 7280 invoked by uid 99); 9 Aug 2011 21:24:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Aug 2011 21:24:44 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of hfyuan@rhapsody.com does not designate 209.85.210.174 as permitted sender) Received: from [209.85.210.174] (HELO mail-iy0-f174.google.com) (209.85.210.174) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Aug 2011 21:24:35 +0000 Received: by iyf40 with SMTP id 40so728381iyf.33 for ; Tue, 09 Aug 2011 14:24:15 -0700 (PDT) Received: by 10.231.113.170 with SMTP id a42mr8193643ibq.64.1312925054976; Tue, 09 Aug 2011 14:24:14 -0700 (PDT) Received: from cafebabe.sea1.rhapsody.com (rev-66-150-171-4.rhapsody.com [66.150.171.4]) by mx.google.com with ESMTPS id v16sm195156ibe.0.2011.08.09.14.24.13 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 09 Aug 2011 14:24:14 -0700 (PDT) From: Hefeng Yuan Content-Type: multipart/alternative; boundary="Apple-Mail=_52E53A85-6BA9-4E2E-AC90-A186F19CBE6F" Subject: AnitEntropy/MerkleTree Error Date: Tue, 9 Aug 2011 14:24:26 -0700 Message-Id: To: user@cassandra.apache.org Mime-Version: 1.0 (Apple Message framework v1244.3) X-Mailer: Apple Mail (2.1244.3) --Apple-Mail=_52E53A85-6BA9-4E2E-AC90-A186F19CBE6F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, We are seeing some AntiEntropy error in our production servers, since = it's hard to repro in other env, pasting the stack track here for some = clue.=20 We're using a cluster of 2 data centers, 9 nodes, 6 for online traffic, = 3 for Brisk BI. Our RF is Cassandra:5, Brisk:1. The exception stack is appended in the end. Any help is appreciated. Thanks, Hefeng ERROR [AntiEntropyStage:2] 2011-08-08 04:24:39,556 = AbstractCassandraDaemon.java (line 113) Fatal exception in thread = Thread[AntiEntropyStage:2,5,main] java.lang.AssertionError at = org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.difference(MerkleTree.java:244) at = org.apache.cassandra.service.AntiEntropyService$Differencer.run(AntiEntrop= yService.java:458) at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886) at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908) at java.lang.Thread.run(Thread.java:619) ERROR [AntiEntropyStage:1] 2011-08-08 04:24:39,580 = AbstractCassandraDaemon.java (line 113) Fatal exception in thread = Thread[AntiEntropyStage:1,5,main] java.lang.AssertionError at = org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= ) at = org.apache.cassandra.utils.MerkleTree.difference(MerkleTree.java:244) at = org.apache.cassandra.service.AntiEntropyService$Differencer.run(AntiEntrop= yService.java:458) at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886) at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908) at java.lang.Thread.run(Thread.java:619)= --Apple-Mail=_52E53A85-6BA9-4E2E-AC90-A186F19CBE6F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

We = are seeing some AntiEntropy error in our production servers, since it's = hard to repro in other env, pasting the stack track here for some = clue. 

We're = using a cluster of 2 data centers, 9 nodes, 6 for online traffic, 3 for = Brisk BI. Our RF is Cassandra:5, Brisk:1.
The exception stack is appended in the = end.

Any = help is appreciated.

Hefeng


ERROR [AntiEntropyStage:2] = 2011-08-08 04:24:39,556 AbstractCassandraDaemon.java (line 113) Fatal = exception in thread Thread[AntiEntropyStage:2,5,main]
at = org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154)
at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.difference(MerkleTree.java:244)
at = org.apache.cassandra.service.AntiEntropyService$Differencer.run(AntiEntrop= yService.java:458)
at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)
at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908)
at = java.lang.Thread.run(Thread.java:619)
ERROR [AntiEntropyStage:1] = 2011-08-08 04:24:39,580 AbstractCassandraDaemon.java (line 113) Fatal = exception in thread Thread[AntiEntropyStage:1,5,main]
at = org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154)
at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273= )
= at = org.apache.cassandra.utils.MerkleTree.difference(MerkleTree.java:244)
at = org.apache.cassandra.service.AntiEntropyService$Differencer.run(AntiEntrop= yService.java:458)
at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)
at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908)
at = java.lang.Thread.run(Thread.java:619)
= --Apple-Mail=_52E53A85-6BA9-4E2E-AC90-A186F19CBE6F--