Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rene.kochen@schange.com
 designates 209.85.215.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+LRpphfV_GVr98KLuxkqD+zKJHgvXxyDOGzwrbG254C1HNhgQ@mail.gmail.com>
References: 
 <CA+LRpphfV_GVr98KLuxkqD+zKJHgvXxyDOGzwrbG254C1HNhgQ@mail.gmail.com>
Date: Fri, 28 Jun 2013 17:14:53 +0200
Message-ID: 
 <CA+LRppjqDciszcMW8qZMfuHX6OZp2FttwnyXYXwSnrYh6RUrSA@mail.gmail.com>
Subject: Re: Cluster processing not balanced
From: Rene Kochen <rene.kochen@schange.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a113403d83db03204e0385701

--001a113403d83db03204e0385701
Content-Type: text/plain; charset=ISO-8859-1

I found the issue.

Stupid mistake.

Although I have 1000000 customers in the cluster, I do a get_range_slices
to get the first 10000 customers. I use these customers as a seed for my
test program. I random select customers from that set and fire requests at
the cluster. Those 10000 customers all reside on the first node (actually
the first three nodes because of the replication factor of three).

Thanks,

Rene


2013/6/28 Rene Kochen <rene.kochen@schange.com>

> Hi All,
>
> I have the following situation:
>
> - Cassandra 1.0.11
> - A 6 node cluster
> - Random partitioner
> - Tokens are balanced (according to node-tool)
> - Data-load is balanced (according to node-tool)
>
> I have a customers column-family with 1000000 customers. I also have a
> test client which requests random customers from the cluster in a
> round-robin fashion, i.e. each request is targeted to a different machine
> and each request contains a random customer-id (a number in the range of 0
> - 1000000).
>
> What I see is that one node (node 4) is only proxying. Node 4 does receive
> requests. I can see that in the storage-proxy counters. However, I see in
> the Customers column-family a zero in the ReadCount counter (local reads).
> Node 4 should also serve reads because I have 1000000 customers in the
> cluster, the cluster is balanced and I do requests with random
> customer-ids. In fact, two of the four nodes have this behavior. The result
> is that the processing load within the cluster is not balanced.
>
> I have never witnessed this behavior before.
>
> Am I missing something?
>
> Thanks,
>
> Rene
>

--001a113403d83db03204e0385701
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I found the issue.<br><br>Stupid mistake.<br><br>Although =
I have 1000000 customers in the cluster, I do a get_range_slices to get the=
 first 10000 customers. I use these customers as a seed for my test program=
. I random select customers from that set and fire requests at the cluster.=
 Those 10000 customers all reside on the first node (actually the first thr=
ee nodes because of the replication factor of three).<br>
<br>Thanks,<br><br>Rene<br></div><div class=3D"gmail_extra"><br><br><div cl=
ass=3D"gmail_quote">2013/6/28 Rene Kochen <span dir=3D"ltr">&lt;<a href=3D"=
mailto:rene.kochen@schange.com" target=3D"_blank">rene.kochen@schange.com</=
a>&gt;</span><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi All,<br><br>I have the f=
ollowing situation:<br><br>- Cassandra 1.0.11<br>- A 6 node cluster<br>- Ra=
ndom partitioner<br>
- Tokens are balanced (according to node-tool)<br>- Data-load is balanced (=
according to node-tool)<br>
<br>I have a customers column-family with 1000000 customers. I also have a =
test client which requests random customers from the cluster in a round-rob=
in fashion, i.e. each request is targeted to a different machine and each r=
equest contains a random customer-id (a number in the range of 0 - 1000000)=
.<br>

<br>What I see is that one node (node 4) is only proxying. Node 4 does rece=
ive requests. I can see that in the storage-proxy counters. However, I see =
in the Customers column-family a zero in the ReadCount counter (local reads=
). Node 4 should also serve reads because I have 1000000 customers in the c=
luster, the cluster is balanced and I do requests with random customer-ids.=
 In fact, two of the four nodes have this behavior. The result is that the =
processing load within the cluster is not balanced.<br>

<br>I have never witnessed this behavior before.<br><br>Am I missing someth=
ing?<br><br>Thanks,<br><br>Rene<br></div>
</blockquote></div><br></div>

--001a113403d83db03204e0385701--