Return-Path: X-Original-To: apmail-cassandra-dev-archive@www.apache.org Delivered-To: apmail-cassandra-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 99FE4E437 for ; Fri, 18 Jan 2013 00:18:00 +0000 (UTC) Received: (qmail 42173 invoked by uid 500); 18 Jan 2013 00:17:59 -0000 Delivered-To: apmail-cassandra-dev-archive@cassandra.apache.org Received: (qmail 42140 invoked by uid 500); 18 Jan 2013 00:17:59 -0000 Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list dev@cassandra.apache.org Received: (qmail 42132 invoked by uid 99); 18 Jan 2013 00:17:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2013 00:17:59 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.210.45 as permitted sender) Received: from [209.85.210.45] (HELO mail-da0-f45.google.com) (209.85.210.45) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2013 00:17:53 +0000 Received: by mail-da0-f45.google.com with SMTP id w4so1351944dam.32 for ; Thu, 17 Jan 2013 16:17:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=sf7ZS33b7xwssj99OG14BGv+KEVzymCEIxiJE3rNfiQ=; b=JxzYtaJ/gG6KOCNrZi17IXTTmo95vc369Kj3EoWHLmXIx+gNPOCZatunIP3r04rjMl biXMu7VGVakrSOZR2CK0xumnsfVbnmxCgMOyTd5LrOav8voYzArPYaRoiF5yUhMKGVlB qvGABz0OOjNIFp3IffsdwnrugF+9mLSmm18F6v+XV7SOiOJh2NK4yc8JxR4UU79lFO79 jacB6NrnzFDiFeqntsUa8PRIjgjUS/4tVXDkMQTby/urNBI9QFOgb/X+nsq0bsPQHCId QmfOz0k12J4rxgyIJHXKZj08DeFqumQlC8AQh5+ueusNm7o1Ad6X7i14ONLPuomOpv61 AvFA== X-Received: by 10.68.143.162 with SMTP id sf2mr18318991pbb.137.1358468252399; Thu, 17 Jan 2013 16:17:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.67.5.69 with HTTP; Thu, 17 Jan 2013 16:17:12 -0800 (PST) In-Reply-To: <1358466286.18699.140661179049785.2AD3CAC2@webmail.messagingengine.com> References: <1358466286.18699.140661179049785.2AD3CAC2@webmail.messagingengine.com> From: Jonathan Ellis Date: Thu, 17 Jan 2013 18:17:12 -0600 Message-ID: Subject: Re: ConcurrentModificationException on boot To: dev@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org You're right. Created CASSANDRA-5170 and attached a patch switching to ConcurrentHashMap. On Thu, Jan 17, 2013 at 5:44 PM, Kirk True wrote: > Hi all, > > > > A client is using Cassandra 1.2.0 (embedded) into their system. On > startup of one of the nodes in the system, they're seeing a > ConcurrentModificationException: > > > > java.util.ConcurrentModificationException > > at java.util.HashMap$HashIterator.nextEntry(HashMap.java:810) > > at java.util.HashMap$KeyIterator.next(HashMap.java:845) > > at > org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.j > ava:146) > > at > org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.jav > a:135) > > at > org.apache.cassandra.dht.BootStrapper.getBootstrapTokens(BootStrapper.j > ava:115) > > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageServic > e.java:611) > > at > org.apache.cassandra.service.StorageService.initServer(StorageService.j > ava:499) > > at > org.apache.cassandra.service.StorageService.initServer(StorageService.j > ava:397) > > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java > :309) > > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.j > ava:397) > > at CassandraServer.start(CassandraServer.java:415) > > > > LoadBroadcaster has an internal map named loadInfo that it both > modifies internally as well as makes available via an accessor method > (getLoadInfo). > > > > StorageService.joinTokenRing contains this line (611): > > > > tokens = BootStrapper.getBootstrapTokens(tokenMetadata, > LoadBroadcaster.instance.getLoadInfo()); > > > > So that's how BootStrapper gets access to LoadBroadcaster's loadInfo. > > > > There are a few places where LoadBroadcaster is called by Gossiper to > add and remove nodes which then modifies loadInfo. There's no logging > in many of those places, so it's hard to know which one it is. > > > > Should LoadBroadcaster.instance.getLoadInfo() return a copy of the > internal map rather than just a reference to it? > > > > Thanks, > > Kirk -- Jonathan Ellis Project Chair, Apache Cassandra co-founder, http://www.datastax.com @spyced