Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 16532 invoked from network); 28 Oct 2010 11:36:49 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 28 Oct 2010 11:36:49 -0000 Received: (qmail 72849 invoked by uid 500); 28 Oct 2010 11:36:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 72836 invoked by uid 500); 28 Oct 2010 11:36:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72681 invoked by uid 99); 28 Oct 2010 11:36:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 11:36:47 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 11:36:42 +0000 Received: by iwn40 with SMTP id 40so2270734iwn.31 for ; Thu, 28 Oct 2010 04:36:21 -0700 (PDT) Received: by 10.231.169.70 with SMTP id x6mr1100562iby.2.1288265781059; Thu, 28 Oct 2010 04:36:21 -0700 (PDT) Received: from mail-iw0-f172.google.com (mail-iw0-f172.google.com [209.85.214.172]) by mx.google.com with ESMTPS id u6sm2349725ibd.12.2010.10.28.04.36.20 (version=SSLv3 cipher=RC4-MD5); Thu, 28 Oct 2010 04:36:20 -0700 (PDT) Received: by iwn40 with SMTP id 40so2270711iwn.31 for ; Thu, 28 Oct 2010 04:36:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.35.11 with SMTP id n11mr9738572ibd.168.1288265779780; Thu, 28 Oct 2010 04:36:19 -0700 (PDT) Received: by 10.220.163.84 with HTTP; Thu, 28 Oct 2010 04:36:19 -0700 (PDT) In-Reply-To: References: Date: Thu, 28 Oct 2010 13:36:19 +0200 Message-ID: Subject: Re: Cluster load balancing? From: Thibaut Britz To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000325576e322e1f2c0493abc05c --000325576e322e1f2c0493abc05c Content-Type: text/plain; charset=ISO-8859-1 Yes. I even tried just starting one node only, and then bootstrapping another node. (However, at the beginning a few days ago, the cluster was unstable and unresponsive and I had to restart the cluster. Maybe something went wrong back then.) Anyway, I will export all the data, and reimport it with the new randomparitioner, which I should have used from the beginning. Thanks, Thibaut On Thu, Oct 28, 2010 at 12:35 AM, Tyler Hobbs wrote: > Not sure if this is the cause, but do all of your nodes have the same seed > list? Did you bring up the seeds first? > > - Tyler > > > On Wed, Oct 27, 2010 at 1:46 PM, Thibaut Britz < > thibaut.britz@trendiction.com> wrote: > >> Depending on the range I choose, choosing manually a token will also fail. >> (node will never exit boostrap, streams doesn't list any open streams) >> >> >> INFO [Thread-53] 2010-10-27 20:33:37,399 SSTableReader.java (line 120) >> Sampling index for /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db >> INFO [Thread-53] 2010-10-27 20:33:37,444 StreamCompletionHandler.java >> (line 64) Streaming added /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db >> >> Stacktracke: >> >> "pool-1-thread-53" prio=10 tid=0x00000000412f2800 nid=0x215c runnable >> [0x00007fd7cf217000] >> java.lang.Thread.State: RUNNABLE >> at java.net.SocketInputStream.socketRead0(Native Method) >> at java.net.SocketInputStream.read(SocketInputStream.java:129) >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) >> at java.io.BufferedInputStream.read(BufferedInputStream.java:317) >> - locked <0x00007fd7e77e0520> (a java.io.BufferedInputStream) >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:126) >> at >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192) >> at >> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1154) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> >> >> >> >> >> >> On Wed, Oct 27, 2010 at 8:27 PM, Thibaut Britz < >> thibaut.britz@trendiction.com> wrote: >> >>> Hello Tyler, >>> >>> thanksf or the quick answer. That's true, I should have noticed. >>> >>> I also tried kicking out one node, clearing all directories and then >>> restarting it with the bootstrap option. It received a few files, but just >>> set there in bootstrapping mode (streams always printed bootstrapping >>> without any files open), forever (> 15 minutes). I stopped the applicaiton >>> so it couldn't be load related, and also tried with a fresh cluster restart. >>> What could cause this? >>> >>> (This should ahve the advantage of cassandra choosing a key in my range >>> which splits the range evenly?) >>> >>> >>> Thanks, >>> Thibaut >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Oct 27, 2010 at 7:40 PM, Tyler Hobbs wrote: >>> >>>> With OrderPreservingPartitioner, you have to keep the ring balanced >>>> manually. >>>> This is why people frequently suggest that you use RandomPartitioner >>>> unless >>>> you absolutely have to do otherwise. With OPP, keys are *not* evenly >>>> distributed >>>> around the ring. >>>> >>>> Apparently you have lots of keys that are between ~'t' and 'x', so start >>>> bunching >>>> your tokens there. >>>> >>>> - Tyler >>>> >>>> >>>> On Wed, Oct 27, 2010 at 12:00 PM, Thibaut Britz < >>>> thibaut.britz@trendiction.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have a little java hector test application running whcih writes and >>>>> reads data to my little cassandra cluster (7 nodes). >>>>> >>>>> The data doesn't get loadbalanced at all: >>>>> >>>>> 192.168.1.12 Up 178.32 MB >>>>> 8S6VvT7oKNcQTso3 |<--| >>>>> 192.168.1.14 Up 30.12 MB >>>>> 9tybk3nB6JCtqQU1 | ^ >>>>> 192.168.1.15 Up 11.96 MB >>>>> RZVG3NC3ksqjEmYE v | >>>>> 192.168.1.16 Up 668.7 KB >>>>> aTV6W12YxxMI31Z8 | ^ >>>>> 192.168.1.10 Up 22.86 GB >>>>> u5iaQxEfyUSwnPn1 v | >>>>> 192.168.1.13 Up 22.5 GB >>>>> vZlWeU8b6LBeAcAY | ^ >>>>> 192.168.1.11 Up 22.27 GB >>>>> xrmaUS6nnrYFSk8e |-->| >>>>> >>>>> What could be the issue? I couldn't find anything in the FAQ related to >>>>> this >>>>> >>>>> Will data (writes) always be added to the server I connect to? If so, >>>>> why will the replicas then always be stored on the same 2 other machines. >>>>> >>>>> (Tested with >>>>> org.apache.cassandra.dht.OrderPreservingPartitioner >>>>> on 0.6.5 and replication level 3) >>>>> >>>>> Thanks, >>>>> Thibaut >>>>> >>>>> >>>>> >>>>> >>>> >>> >> > --000325576e322e1f2c0493abc05c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Yes. I even tried just starting one node only, and then bootstrapping anoth= er node. (However, at the beginning a few days ago, the cluster was unstabl= e and unresponsive and I had to restart the cluster. Maybe something went w= rong back then.)

Anyway, I will export all the data, and reimport it with the new random= paritioner, which I should have used from the beginning.

Thanks,
= Thibaut






On Thu, Oct 2= 8, 2010 at 12:35 AM, Tyler Hobbs <tyler@riptano.com> wrote:
Not sure if this = is the cause, but do all of your nodes have the same seed list?=A0 Did you = bring up the seeds first?

- Tyler


On Wed, Oct 27, 2010 at 1:46 PM, Thibau= t Britz <thibaut.britz@trendiction.com> wrote:
Depending on the = range I choose, choosing manually a token will also fail. (node will never = exit boostrap, streams doesn't list any open streams)


=A0INFO [Thread-53] 2010-10-27 20:33:37,399 SSTableReader.java (lin= e 120) Sampling index for /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db=
=A0INFO [Thread-53] 2010-10-27 20:33:37,444 StreamCompletionHandler.java (l= ine 64) Streaming added /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db
Stacktracke:

"pool-1-thread-53" prio=3D10 tid=3D0x00= 000000412f2800 nid=3D0x215c runnable [0x00007fd7cf217000]
=A0=A0 java.lang.Thread.State: RUNNABLE
=A0=A0=A0=A0=A0=A0=A0 at java.ne= t.SocketInputStream.socketRead0(Native Method)
=A0=A0=A0=A0=A0=A0=A0 at = java.net.SocketInputStream.read(SocketInputStream.java:129)
=A0=A0=A0=A0= =A0=A0=A0 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)=
=A0=A0=A0=A0=A0=A0=A0 at java.io.BufferedInputStream.read1(BufferedInputStr= eam.java:258)
=A0=A0=A0=A0=A0=A0=A0 at java.io.BufferedInputStream.read(= BufferedInputStream.java:317)
=A0=A0=A0=A0=A0=A0=A0 - locked <0x00007= fd7e77e0520> (a java.io.BufferedInputStream)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.transport.TIOStreamTransport.rea= d(TIOStreamTransport.java:126)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrif= t.transport.TTransport.readAll(TTransport.java:84)
=A0=A0=A0=A0=A0=A0=A0= at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java= :314)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.protocol.TBinaryProtocol.readI32= (TBinaryProtocol.java:262)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.pr= otocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
=A0=A0= =A0=A0=A0=A0=A0 at org.apache.cassandra.thrift.Cassandra$Processor.process(= Cassandra.java:1154)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.thrift.CustomTThreadPoolServe= r$WorkerProcess.run(CustomTThreadPoolServer.java:167)
=A0=A0=A0=A0=A0=A0= =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe= cutor.java:886)
=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPool= Executor$Worker.run(ThreadPoolExecutor.java:908)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)






On Wed, Oct 27, = 2010 at 8:27 PM, Thibaut Britz <thibaut.britz@trendiction.com<= /a>> wrote:
Hello Tyler,
<= br>thanksf or the quick answer. That's true, I should have noticed.

I also tried kicking out one node, clearing all directories and then re= starting it with the bootstrap option. It received a few files, but just se= t there in bootstrapping mode (streams always printed bootstrapping without= any files open), forever (> 15 minutes). I stopped the applicaiton so i= t couldn't be load related, and also tried with a fresh cluster restart= . What could cause this?

(This should ahve the advantage of cassandra choosing a key in my range= which splits the range evenly?)


Thanks,
Thibaut








On Wed, Oct 27, 2010 at 7:40 PM, Tyler H= obbs <tyler@riptano.com> wrote:
With OrderPreserv= ingPartitioner, you have to keep the ring balanced manually.
This is why= people frequently suggest that you use RandomPartitioner unless
you absolutely have to do otherwise.=A0 With OPP, keys are *not* evenly dis= tributed
around the ring.

Apparently you have lots of keys that are between ~= 't' and 'x', so start bunching
your tokens there.

- Tyler


On Wed, Oct 27, 2010 at 12:00 PM, Thibaut Br= itz <thibaut.britz@trendiction.com> wrote:
Hi,

I have= a little java hector test application running whcih writes and reads data = to my little cassandra cluster (7 nodes).

The data doesn't get loadbalanced at all:

192.168.1.12 Up=A0= =A0=A0=A0=A0=A0=A0=A0 178.32 MB=A0=A0=A0=A0 8S6VvT7oKNcQTso3=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |<--| 192.168.1.14 Up=A0=A0=A0=A0=A0=A0=A0=A0 30.12 MB=A0=A0=A0=A0=A0 9tybk3nB6JC= tqQU1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 |=A0=A0 ^
192.168.1.15 Up=A0=A0=A0=A0=A0=A0=A0=A0 11.96 MB=A0= =A0=A0=A0=A0 RZVG3NC3ksqjEmYE=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 v=A0=A0 |
192.168.1.16 Up=A0=A0=A0=A0= =A0=A0=A0=A0 668.7 KB=A0=A0=A0=A0=A0 aTV6W12YxxMI31Z8=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |=A0=A0 ^
192.168.1.10 Up=A0=A0=A0=A0=A0=A0=A0=A0 22.86 GB=A0=A0=A0=A0=A0 u5iaQxEfyUS= wnPn1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 v=A0=A0 |
192.168.1.13 Up=A0=A0=A0=A0=A0=A0=A0=A0 22.5 GB=A0= =A0=A0=A0=A0=A0 vZlWeU8b6LBeAcAY=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |=A0=A0 ^
192.168.1.11 Up=A0=A0=A0= =A0=A0=A0=A0=A0 22.27 GB=A0=A0=A0=A0=A0 xrmaUS6nnrYFSk8e=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |-->|

What could be the issue? I couldn't find anything in the FAQ relate= d to this

Will data (writes) always be added to the server I connect= to? If so, why will the replicas then always be stored on the same 2 other= machines.

(Tested with <Partitioner>org.apache.cassandra.dht.OrderPreservin= gPartitioner</Partitioner> on 0.6.5 and replication level 3)

T= hanks,
Thibaut








--000325576e322e1f2c0493abc05c--