From user-return-59395-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Mon Jan 15 06:18:47 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 47D29180651 for ; Mon, 15 Jan 2018 06:18:47 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3826D160C44; Mon, 15 Jan 2018 05:18:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 56D1B160C43 for ; Mon, 15 Jan 2018 06:18:46 +0100 (CET) Received: (qmail 68697 invoked by uid 500); 15 Jan 2018 05:18:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 68680 invoked by uid 99); 15 Jan 2018 05:18:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jan 2018 05:18:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9DCE51A077E for ; Mon, 15 Jan 2018 05:18:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.979 X-Spam-Level: * X-Spam-Status: No, score=1.979 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=instaclustr-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id MGXZNH4Ob1qz for ; Mon, 15 Jan 2018 05:18:40 +0000 (UTC) Received: from mail-yw0-f177.google.com (mail-yw0-f177.google.com [209.85.161.177]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 200FC5F295 for ; Mon, 15 Jan 2018 05:18:40 +0000 (UTC) Received: by mail-yw0-f177.google.com with SMTP id b129so4957108ywa.8 for ; Sun, 14 Jan 2018 21:18:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=instaclustr-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Gp64099WLCGykJy30fKDRGgm7E1G1/Vuy3yEmSsIXA4=; b=FRh+RprByL8lzLNt422MGM8PYTkSXg1dvB8UItxOObqkYxxqhPA3o/CM5va4/CfH3r Y6sRlA4SsiNwWEgPz3W55ETJ+XgQSB2qkazhivrr1+cNVMjSN/eKyjzU8zCt5DxLhDms pibNbq8CKm4s5dlR6CYvsDzwsOKB3nFtmCgVS0mtDRjo5dX3ry05VdcXPW53qj8UXevt S8xaJTr0tTT2G6Dr7jhWSX8qc8wzf8PVdj2OOlbi2bhPaPAnf8gqz9eozQzQEcIuDKxD wzFsbfICs8TZdsA5gnIq83FtDyLUlXKMcXhxfi4DrBwIjS0fuB343xeTvzIHnajYYcND nVxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Gp64099WLCGykJy30fKDRGgm7E1G1/Vuy3yEmSsIXA4=; b=G6Z+qu5q/AVNA/iZKS2JNJBAF/UXWl/EiZq/bgnBuig6GIFV4clMAfprGbbmUOLQJf C8vXeJweS3v7kUa1LKHkqJt1ZCT+InlRe5Hxkf9F6SCf6zhTyBx/Wp2SK7ILrsYICWO5 VlgV2fxcbBNQL1GYwT5KOonYtfir/W8jTydFdoNw2zy+wbhjpLqE49V168ysvaccwRJe jhnnczoZx7KXqGkzMRzwptIGhdcUumyJDbgoIKfbc/v8XuRTEuqYRBgfoO5T45pZB+Zg GnrUyb/CGi6UJIbagA5/yu1da6d9L4qkDSduQVI+YuggcxZAW7rL4qVA/e6A6wndKYQY tQaw== X-Gm-Message-State: AKwxytdD1rC8PHObtlLtWX08zyIr13rGa8INKzEDecwj7LYNBVmysgsG ABizUmTGBcnEa1r5GjOzuNPQp/mJBd63qCDBXci/ggd7pmQ= X-Google-Smtp-Source: ACJfBosPssKGjJ82UZlElzjLHHbUGLTViPyn7d+l7+KTv7RiDqY8VLpYAw5quVZ43ePVHfVKser8zgQ6ETOkAaepNTo= X-Received: by 10.129.98.197 with SMTP id w188mr20319324ywb.74.1515993518908; Sun, 14 Jan 2018 21:18:38 -0800 (PST) MIME-Version: 1.0 Received: by 10.129.83.67 with HTTP; Sun, 14 Jan 2018 21:17:58 -0800 (PST) In-Reply-To: References: From: kurt greaves Date: Mon, 15 Jan 2018 05:17:58 +0000 Message-ID: Subject: Re: Cleanup blocking snapshots - Options? To: User Content-Type: multipart/alternative; boundary="001a11470a642d3c5f0562c9bf61" --001a11470a642d3c5f0562c9bf61 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Disabling the snapshots is the best and only real option other than upgrading at the moment. Although apparently it was thought that there was only a small race condition in 2.1 that triggered this and it wasn't worth fixing. If you are triggering it easily maybe it is worth fixing in 2.1 as well. Does this happen consistently? Can you provide some more logs on the JIRA or better yet a way to reproduce? On 14 January 2018 at 16:12, Steinmaurer, Thomas < thomas.steinmaurer@dynatrace.com> wrote: > Hello, > > > > we are running 2.1.18 with vnodes in production and due to ( > https://issues.apache.org/jira/browse/CASSANDRA-11155) we can=E2=80=99t r= un > cleanup e.g. after extending the cluster without blocking our hourly > snapshots. > > > > What options do we have to get rid of partitions a node does not own > anymore? > > =C2=B7 Using a version which has this issue fixed, although upgra= ding > to 2.2+, due to various issues, is not an option at the moment > > =C2=B7 Temporarily disabling the hourly cron job before starting > cleanup and re-enable after cleanup has finished > > =C2=B7 Any other way to re-write SSTables with data a node owns a= fter > a cluster scale out > > > > Thanks, > > Thomas > > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freist > > =C3=A4dterstra > > =C3=9Fe 313 > > --001a11470a642d3c5f0562c9bf61 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Disabling the snapshots is the best and only real option o= ther than upgrading at the moment. Although apparently it was thought that = there was only a small race condition in 2.1 that triggered this and it was= n't worth fixing. If you are triggering it easily maybe it is worth fix= ing in 2.1 as well. Does this happen consistently? Can you provide some mor= e logs on the JIRA or better yet a way to reproduce?

On 14 January 2018 at 16:12, Stein= maurer, Thomas <thomas.steinmaurer@dynatrace.com> wrote:

Hello,

=C2=A0

we are running 2.1.18 with vnod= es in production and due to (https://issues.apache.org/jira/b= rowse/CASSANDRA-11155) we can=E2=80=99t run cleanup e.g. after extendin= g the cluster without blocking our hourly snapshots.

=C2=A0

What options do we have to get = rid of partitions a node does not own anymore?

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Using a version which has = this issue fixed, although upgrading to 2.2+, due to various issues, is not= an option at the moment

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Temporarily disabling the = hourly cron job before starting cleanup and re-enable after cleanup has fin= ished

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Any other way to re-write = SSTables with data a node owns after a cluster scale out

=C2=A0

Thanks,

Thomas

=C2=A0

The contents of this e-mail are intended for the named addressee only. It c= ontains information that may be confidential. Unless you are the named addr= essee or an authorized designee, you may not copy or use it, or disclose it= to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Au= stria GmbH (registration number FN 91482h) is a company registered in Linz = whose registered office is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313

--001a11470a642d3c5f0562c9bf61--