Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E2C88200C16 for ; Thu, 9 Feb 2017 09:54:13 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E1954160B50; Thu, 9 Feb 2017 08:54:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DDB0B160B4C for ; Thu, 9 Feb 2017 09:54:12 +0100 (CET) Received: (qmail 67273 invoked by uid 500); 9 Feb 2017 08:54:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 67262 invoked by uid 99); 9 Feb 2017 08:54:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Feb 2017 08:54:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C5954C03A2 for ; Thu, 9 Feb 2017 08:54:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.649 X-Spam-Level: ** X-Spam-Status: No, score=2.649 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id UGrBdmGc-AzO for ; Thu, 9 Feb 2017 08:54:07 +0000 (UTC) Received: from mail-pf0-f172.google.com (mail-pf0-f172.google.com [209.85.192.172]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A57DA5FC5F for ; Thu, 9 Feb 2017 08:54:06 +0000 (UTC) Received: by mail-pf0-f172.google.com with SMTP id e4so48890546pfg.1 for ; Thu, 09 Feb 2017 00:54:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=4gMX/AFn3MI3oiP8srK2cfeDxZneMFJCgYm1ltB0EVQ=; b=KPnKQQ1/NF8ahIN0TWwK3zjCVumjwBux+EIYI4GhCZ8uEMgv1EuNspPri5VWA0YS3r f+vuYDuxq4KfaDI91fJ4tGvehpuRWOn2vWmKZj2HQ107AzfKeHu03UZeG3i5sKo662Rn 6cuIfsBn02V3jY2ygBnZDpAVLf/ZhNEBEEGMaouRx07y2p9+g0skxDvEodsSL+1foxqI Y9g4JtmiIp4XW8tBiheNEvCXXuIWlh52YtgnGzgPOWzqx1TMw7/OZWLVmXPlNOp4ztBs F2dp3++8HJvXo4Un8eKprtota8cRB3QPDb1Rn4N3kGsjXVpaheQyr0YHBsG6Mtymr28+ W+Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=4gMX/AFn3MI3oiP8srK2cfeDxZneMFJCgYm1ltB0EVQ=; b=LdAr20/zvoyRksYb0seMMwUNRV7tjFhTXrxoTvCtVEFKdqmD4FbaaBkvqNBfy6bM9d ealz6emTN3m5i6OK0F40px+KysixbF30iqOC6U6nl5iDAMnHu2yz7t1UVIcIn0A/qhNq AnCS6XMS+YUfYSaV9XDzEB2KzihR/3AdXd/9XcigYtp3xYBTGa6s2X94nqGGnLD3A9q6 Jf08pCrcU3HDFNp1C3Yy6eAaLdSIbV3d2TwnoshS+D1Yoc4MmaPxllDFQFCLTdTIQSJM PINVdugBp638n8f+2MEDlkxw8Uo3cDEj8cKCnQ1X4/T7qRKwcHmFQU7NOBkzM67oWdsm AFVA== X-Gm-Message-State: AMke39k3y8IGCyg+y1iojnyFROxnS51ySijKQmthrzX5F+w473Q7JQ/gKY1Z02n5FYtcCg== X-Received: by 10.99.99.5 with SMTP id x5mr2592804pgb.225.1486630444971; Thu, 09 Feb 2017 00:54:04 -0800 (PST) Received: from [192.168.1.3] ([183.82.115.243]) by smtp.gmail.com with ESMTPSA id j78sm26673280pfk.39.2017.02.09.00.54.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 09 Feb 2017 00:54:03 -0800 (PST) From: Srinath Reddy Content-Type: multipart/signed; boundary="Apple-Mail=_CF20A8CE-1A57-415E-BCEB-33B5C224D296"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: Error when running nodetool cleanup after adding a new node to a cluster Date: Thu, 9 Feb 2017 14:23:59 +0530 References: <7386EE4A-DABB-4129-8DB7-04A8F9F6ABD4@gmail.com> <1E2592E1-8E5C-452D-BF72-E3033CA15BEB@gmail.com> To: user@cassandra.apache.org In-Reply-To: Message-Id: <29F56CEF-5D22-414E-BB2D-A97B856D520E@gmail.com> X-Mailer: Apple Mail (2.3259) archived-at: Thu, 09 Feb 2017 08:54:14 -0000 --Apple-Mail=_CF20A8CE-1A57-415E-BCEB-33B5C224D296 Content-Type: multipart/alternative; boundary="Apple-Mail=_9107F745-A365-4D7B-A5BE-D103228B5576" --Apple-Mail=_9107F745-A365-4D7B-A5BE-D103228B5576 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Alex, Thanks for reply. I will try the workaround and post an update. Regards, Srinath Reddy > On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin = wrote: >=20 > On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy > wrote: > Hi, >=20 > Trying to re-balacne a Cassandra cluster after adding a new node and = I'm getting this error when running nodetool cleanup. The Cassandra = cluster is running in a Kubernetes cluster. >=20 > Cassandra version is 2.2.8 >=20 > nodetool cleanup > error: io.k8s.cassandra.KubernetesSeedProvider > Fatal configuration error; unable to start server. See log for = stacktrace. > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: = io.k8s.cassandra.KubernetesSeedProvider > Fatal configuration error; unable to start server. See log for = stacktrace. > at = org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescrip= tor.java:676) > at = org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor= .java:119) > at = org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256) > at = org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:2= 62) > at = org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55) > at = org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158) >=20 > Hi, >=20 > =46rom the above stacktrace it looks like you're hitting the following = TODO item: >=20 > = https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b1= 8df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282 = >=20 > That is, nodetool needs to know concurrent_compactors setting's value = before starting cleanup, but doesn't use JMX and tries to parse the = configuration file instead. That fails because your custom SeedProvider = class is not on classpath for nodetool. >=20 > A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be = found by java when running nodetool script, see = https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b1= 8df3ce9/bin/nodetool#L108 = >=20 > Proper fix: get rid of the TODO and really query the value using JMX, = especially since the latest tick-tock release of Cassandra (3.10) added = a way to modify it with JMX. >=20 > -- > Alex --Apple-Mail=_9107F745-A365-4D7B-A5BE-D103228B5576 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Alex,

Thanks for reply.  I will try the workaround and post an = update.

Regards,

Srinath Reddy

On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <oleksandr.shulgin@zalando.de> wrote:

On Thu, Feb 9, 2017 at = 6:13 AM, Srinath Reddy <ksreddy@gmail.com> wrote:
Hi,

Trying = to re-balacne a Cassandra cluster after adding a new node and I'm = getting this error when running nodetool cleanup. The Cassandra cluster is = running in a Kubernetes cluster.

Cassandra version is 2.2.8

nodetool= cleanup
error: = io.k8s.cassandra.KubernetesSeedProvider
Fatal = configuration error; unable to start server.  See log for = stacktrace.
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal = configuration error; unable to start server.  See log for = stacktrace.
at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
 
Hi,

=46rom the above stacktrace it looks like you're hitting the = following TODO item:

https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0= f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282=

That is, nodetool needs = to know concurrent_compactors setting's value before starting cleanup, = but doesn't use JMX and tries to parse the configuration file = instead.  That fails because your custom SeedProvider class is not = on classpath for nodetool.

A workaround: make sure = io.k8s.cassandra.KubernetesSeedProvider can be found by java when = running nodetool script, see https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0= f7886a1e9b18df3ce9/bin/nodetool#L108

Proper fix: = get rid of the TODO and really query the value using JMX, especially = since the latest tick-tock release of Cassandra (3.10) added a way to = modify it with JMX.

--
Alex

= --Apple-Mail=_9107F745-A365-4D7B-A5BE-D103228B5576-- --Apple-Mail=_CF20A8CE-1A57-415E-BCEB-33B5C224D296 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlicLigACgkQ4y+fjRiP5BFW6QCglS0Z1OX9NM6Fll87jBXdScgI mC8AoLXJ2W3YLOhQD4/SteJriSHycmQD =G57D -----END PGP SIGNATURE----- --Apple-Mail=_CF20A8CE-1A57-415E-BCEB-33B5C224D296--