Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9F19A6D40 for ; Thu, 4 Aug 2011 08:56:41 +0000 (UTC) Received: (qmail 83532 invoked by uid 500); 4 Aug 2011 08:56:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81603 invoked by uid 500); 4 Aug 2011 08:56:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81531 invoked by uid 500); 4 Aug 2011 08:56:20 -0000 Delivered-To: apmail-incubator-cassandra-user@incubator.apache.org Received: (qmail 81474 invoked by uid 99); 4 Aug 2011 08:56:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Aug 2011 08:56:18 +0000 X-ASF-Spam-Status: No, hits=3.9 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of springrider@gmail.com designates 209.85.215.47 as permitted sender) Received: from [209.85.215.47] (HELO mail-ew0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Aug 2011 08:56:10 +0000 Received: by ewy5 with SMTP id 5so848704ewy.6 for ; Thu, 04 Aug 2011 01:55:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=rTWmcZ8c9qlrrsEkONlMXypB5Gs4+F5Tvb8cVd7iI0I=; b=jld6IVePG1gSjP2fdFXiiuw1OhrD4DeCG9NsOBa33YJ1NGQSi+EHzCpZoxNf9kjOFm /z27V4Bbw+JhS8CwMRAF3HutduZrrHB2PKj+LCaVsK9Baoi2l7BID6ZlsSEe1C0C+IpV qzNE3J0AvUmGVn0VfKeRQT+hvuDWBlNQ9j3/k= Received: by 10.213.17.74 with SMTP id r10mr185006eba.148.1312448150295; Thu, 04 Aug 2011 01:55:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.35.142 with HTTP; Thu, 4 Aug 2011 01:55:30 -0700 (PDT) In-Reply-To: References: <1312132674399-6638649.post@n2.nabble.com> <1312153659782-6639317.post@n2.nabble.com> From: Yan Chunlu Date: Thu, 4 Aug 2011 16:55:30 +0800 Message-ID: Subject: Re: how to solve one node is in heavy load in unbalanced cluster To: user@cassandra.apache.org Cc: cassandra-user@incubator.apache.org Content-Type: multipart/alternative; boundary=0015174c1402c8d3de04a9aa2574 X-Virus-Checked: Checked by ClamAV on apache.org --0015174c1402c8d3de04a9aa2574 Content-Type: text/plain; charset=ISO-8859-1 I have tried the nodetool move but get the following error.... node3:~# nodetool -h node3 move 0 Exception in thread "main" java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:930) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:896) at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1596) at org.apache.cassandra.service.StorageService.move(StorageService.java:1734) at org.apache.cassandra.service.StorageService.move(StorageService.java:1709) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) then nodetool shows the node is leaving.... nodetool -h reagon ring Address Status State Load Owns Token 84944475733633104818662955375549269696 node3 Up Normal 13.18 GB 81.09% 52773518586096316348543097376923124102 node3 Up Normal 22.85 GB 10.48% 70597222385644499881390884416714081360 node3 Up Leaving 25.44 GB 8.43% 84944475733633104818662955375549269696 the log didn't show any error message neither anything abnormal. is there something wrong? I used to have RF=2, and changed it to RF=3 using cassandra-cli. On Mon, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: > thanks a lot! I will try the "move". > > > On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: > >> >> springrider wrote: >> > >> > is that okay to do nodetool move before a completely repair? >> > >> > using this equation? >> > def tokens(nodes): >> > >> > - for x in xrange(nodes): >> > - print 2 ** 127 / nodes * x >> > >> >> Yes use that logic to get the tokens. I think it's safe to run move first >> and reair later. You are moving some nodes data as is so it's no worse >> than >> what you have right now. >> >> -- >> View this message in context: >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >> Sent from the cassandra-user@incubator.apache.org mailing list archive at >> Nabble.com. >> > > --0015174c1402c8d3de04a9aa2574 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I have tried the nodetool move but get the following error....

node3:~# nodetool -h node3 move 0
Exception in thread= "main" java.lang.IllegalStateException: replication factor (3) e= xceeds number of endpoints (2)
at or= g.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleS= trategy.java:60)
at org.apache.cassandra.service.StorageService.calculatePe= ndingRanges(StorageService.java:930)
at or= g.apache.cassandra.service.StorageService.calculatePendingRanges(StorageSer= vice.java:896)
at org.apache.cassandra.service.StorageService.startLeaving(= StorageService.java:1596)
at or= g.apache.cassandra.service.StorageService.move(StorageService.java:1734)
at= org.apache.cassandra.service.StorageService.move(StorageService.java:1709)=
at su= n.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.Na= tiveMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at su= n.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.= java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at co= m.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntro= spector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invok= eM2(StandardMBeanIntrospector.java:27)
at co= m.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)=
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at co= m.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
= at com.sun= .jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInt= erceptor.java:836)
at co= m.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
<= div> at jav= ax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.ja= va:1427)
at ja= vax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.ja= va:72)
= at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation= .run(RMIConnectionImpl.java:1265)
at ja= vax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnec= tionImpl.java:1360)
at javax.management.remote.rmi.RMIConnectionImpl.invoke= (RMIConnectionImpl.java:788)
at su= n.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.= DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at ja= va.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastSer= verRef.dispatch(UnicastServerRef.java:305)
at su= n.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.Acce= ssController.doPrivileged(Native Method)
at su= n.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transp= ort.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at su= n.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:7= 90)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTranspor= t.java:649)
at ja= va.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.jav= a:886)
= at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec= utor.java:908)
at ja= va.lang.Thread.run(Thread.java:662)




then nodetool shows the node is leaving....<= /div>


nodetool -h reagon ring
A= ddress =A0 =A0 =A0 =A0 Status State =A0 Load =A0 =A0 =A0 =A0 =A0 =A0Owns = =A0 =A0Token =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0=A0
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A084944475= 733633104818662955375549269696 =A0 =A0 =A0
node3 =A0 =A0 =A0Up =A0 =A0 Normal =A013.18 GB =A0 =A0 =A0 =A081.09% = =A052773518586096316348543097376923124102 =A0 =A0 =A0
node3 =A0 = =A0 Up =A0 =A0 Normal =A022.85 GB =A0 =A0 =A0 =A010.48% =A07059722238564449= 9881390884416714081360 =A0 =A0 =A0
node3 =A0 =A0 =A0Up =A0 =A0 Le= aving 25.44 GB =A0 =A0 =A0 =A08.43% =A0 84944475733633104818662955375549269= 696=A0

the log didn't show any error message neither= anything abnormal. =A0is there something wrong?

<= br>
I used to have RF=3D2, and changed it to RF=3D3 using cassand= ra-cli.


On Mon, Aug 1, 2011 at 10:22 = AM, Yan Chunlu <springrider@gmail.com> wrote:
thanks a lot! I will try the "move".


On Mon, Aug 1, 2011 at 7:07 AM, mca= sandra <mohitanchlia@gmail.com> wrote:

springrider wrote:
>
> is that okay to do nodetool move before a completely repair?
>
> using this equation?
> def tokens(nodes):
>
> =A0 =A0- for x in xrange(nodes):
> =A0 =A0 =A0 - print 2 ** 127 / nodes * x
>

Yes use that logic to get the tokens. I think it's safe to run mo= ve first
and reair later. You are moving some nodes data as is so it's no worse = than
what you have right now.

--
View this message in context: http://cassandra= -user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-i= n-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html
Sent from the cassandra-user@incubator.apache.org= mailing list archive at Nabble.com.


--0015174c1402c8d3de04a9aa2574--