Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B8AB9D0D0 for ; Wed, 29 Aug 2012 01:55:51 +0000 (UTC) Received: (qmail 74619 invoked by uid 500); 29 Aug 2012 01:55:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 74569 invoked by uid 500); 29 Aug 2012 01:55:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74561 invoked by uid 99); 29 Aug 2012 01:55:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Aug 2012 01:55:49 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of scode@scode.org designates 209.85.160.44 as permitted sender) Received: from [209.85.160.44] (HELO mail-pb0-f44.google.com) (209.85.160.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Aug 2012 01:55:42 +0000 Received: by pbbrr4 with SMTP id rr4so223495pbb.31 for ; Tue, 28 Aug 2012 18:55:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :x-gm-message-state; bh=E5r9o84DXWOFZSE2JwhESUoUZuIi/ueG6aWcp5H5glI=; b=MIfVtkIDgE4KC/g+CiSIQDl31PjlNjVmMJqB1yDl2+Cs9AfQZpbsrlkVi63KzGjpEf /SfTPB5LQ0nMYIw3qCn3qVnlWEu0VuMCd6QfNIm8XZ2gWJTUaXItXsRg9J3F1XPGDzTl tgSHDs/7jIZCcZJTQrfLmBi8JIur8C+EeDUPKnSOWX09jRHteHYbxhJKgO5ADc8xp0Aq GgVD3sGWNBHzg+ip/Ym8zMX70NFZl1NJ4yzCA4eveRJl8ORL9chl+0Bd2J2UkwnDEtEK ws0s2Whda7eIl/TmaY2VeGuuCThmeE3TfalNv8DYEX4TCYQg9IMI3AbwSM2ESSBIhDJE aAEQ== MIME-Version: 1.0 Received: by 10.68.190.8 with SMTP id gm8mr849505pbc.74.1346205321028; Tue, 28 Aug 2012 18:55:21 -0700 (PDT) Sender: scode@scode.org Received: by 10.66.88.35 with HTTP; Tue, 28 Aug 2012 18:55:20 -0700 (PDT) X-Originating-IP: [71.198.224.174] In-Reply-To: <503D2C5E.8060500@globalrelay.net> References: <50367ACA.4090301@globalrelay.net> <5036937F.8000305@globalrelay.net> <7FB4A3F7-0105-49C5-B630-EABFC611A8FB@thelastpickle.com> <5037C593.6090509@globalrelay.net> <503D2C5E.8060500@globalrelay.net> Date: Tue, 28 Aug 2012 18:55:20 -0700 X-Google-Sender-Auth: 9cdg6AUWHgJFGJ6ZY1zuspdgEHg Message-ID: Subject: Re: Node forgets about most of its column families From: Peter Schuller To: user@cassandra.apache.org Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQmgyH4iV9/f2r3mr6TO4UtRJb0tna3473SfHaNs0J6imLFI4PnDS98P4b1qK5q4ADUa+FnJ X-Virus-Checked: Checked by ClamAV on apache.org I can confirm having seen this (no time to debug). One method of recovery is to jump the node back into the ring with auto_bootstrap set to false and an appropriate token set, after deleting system tables. That assumes you're willing to have the node take a few bad reads until you're able to disablegossip and make other nodes not send requests to it. disabling thrift would also be advised, or even firewalling it prior to restart. -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)