Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B3A609758 for ; Sun, 24 Jun 2012 17:35:20 +0000 (UTC) Received: (qmail 52629 invoked by uid 500); 24 Jun 2012 17:35:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 52595 invoked by uid 500); 24 Jun 2012 17:35:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 52585 invoked by uid 99); 24 Jun 2012 17:35:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jun 2012 17:35:18 +0000 X-ASF-Spam-Status: No, hits=3.6 required=5.0 tests=FSL_FREEMAIL_1,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of safdar.kureishy@gmail.com designates 74.125.82.44 as permitted sender) Received: from [74.125.82.44] (HELO mail-wg0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jun 2012 17:35:10 +0000 Received: by wgbdr13 with SMTP id dr13so2621770wgb.25 for ; Sun, 24 Jun 2012 10:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=BH8OpxiRNLzo2IsEeRGD133Qs9h1LjvSZXi/EX3jV44=; b=hQpXTDEdyn+Pp/ClYu11jxe5Ii/PGty2oLnPegK0MqSbs1EDBZA51k4+zhG1BBnPpg AawEi9xLDeL4Kr/hOVYmUUj/P4FYo0t6ch7tS81D8vk9SKj0P4pkrwzOU/b/rIv4wT1t KOzoxV3ekwldwq+gUDdnieTplMg2s6OIrYff0ZkW5j95aRNRfeSOVti/DswB34HUrKYT kWaU/d9UsvmhoLcdiNbzfhZokthETHqEaoFREzJNkuNdKUUxPkqMzRg4pVLpaPw9UYbx zqYrqAQD8hZVpbW8SbR6AKIguTgHy4xK+mmitSeOEWFQE8SlisYXdqxT2v3gwKezpuEm n8zA== Received: by 10.216.219.11 with SMTP id l11mr4910206wep.213.1340559289940; Sun, 24 Jun 2012 10:34:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.180.5.133 with HTTP; Sun, 24 Jun 2012 10:34:29 -0700 (PDT) Reply-To: safdar.kureishy@gmail.com In-Reply-To: References: From: Safdar Kureishy Date: Sun, 24 Jun 2012 20:34:29 +0300 Message-ID: Subject: Re: RandomPartitioner is providing a very skewed distribution of keys across a 5-node Solandra cluster To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e6dab06446fc2304c33b484e --0016e6dab06446fc2304c33b484e Content-Type: text/plain; charset=ISO-8859-1 An additional detail is that the CPU utilization on those nodes is proportional to the load below, so machines 9.9.9.1 and 9.9.9.3 experience a fraction of CPU load as compared to the remaining 3 nodes. This might further point to the possibility that the keys are hashing minimally to the token ranges on those nodes. I'm no expert at cryptography, but is it possible that web URLs are not evenly distributed via MD5 hashing due to the common prefixes they contain? (such as the "http://" prefix, or perhaps a domain name?)? What's also interesting is that the distribution is more-or-less even across *alternating* nodes... (0, 2, 4 -- vs -- 1, 3). Thanks, Safdar On Sun, Jun 24, 2012 at 6:00 PM, Safdar Kureishy wrote: > Hi, > > I've searched online but was unable to find any leads for the problem > below. This mailing list seemed the most appropriate place. Apologies in > advance if that isn't the case. > > I'm running a 5-node Solandra cluster (Solr + Cassandra). I've setup the > nodes with tokens *evenly distributed across the token space*, for a > 5-node cluster (as evidenced below under the "effective-ownership" column > of the "nodetool ring" output). My data is a set of a few million crawled > web pages, crawled using Nutch, and also indexed using the "solrindex" > command available through Nutch. AFAIK, the key for each document generated > from the crawled data is the URL. > > Based on the "load" values for the nodes below, despite adding about 3 > million web pages to this index via the HTTP Rest API (e.g.: > http://9.9.9.x:8983/solandra/index/update....), some nodes are still > "empty". Specifically, nodes 9.9.9.1 and 9.9.9.3 have just a few kilobytes > (shown in *bold* below) of the index, while the remaining 3 nodes are > consistently getting hammered by all the data. If the RandomPartioner > (which is what I'm using for this cluster) is supposed to achieve an even > distribution of keys across the token space, why is it that the data below > is skewed in this fashion? Literally, no key was yet been hashed to the > nodes 9.9.9.1 and 9.9.9.3 below. Could someone possibly shed some light on > this absurdity?. > > [me@hm1 solandra-app]$ bin/nodetool -h hm1 ring > Address DC Rack Status State Load > Effective-Owership Token > > 136112946768375385385349842972707284580 > 9.9.9.0 datacenter1 rack1 Up Normal 7.57 GB > 20.00% 0 > 9.9.9.1 datacenter1 rack1 Up Normal *21.44 KB* > 20.00% 34028236692093846346337460743176821145 > 9.9.9.2 datacenter1 rack1 Up Normal 14.99 GB > 20.00% 68056473384187692692674921486353642290 > 9.9.9.3 datacenter1 rack1 Up Normal *50.79 KB* > 20.00% 102084710076281539039012382229530463435 > 9.9.9.4 datacenter1 rack1 Up Normal 15.22 GB > 20.00% 136112946768375385385349842972707284580 > > Thanks in advance. > > Regards, > Safdar > --0016e6dab06446fc2304c33b484e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
An additional detail is that the CPU utilization on those = nodes is proportional to the load below, so machines 9.9.9.1 and 9.9.9.3 ex= perience a fraction of CPU load as compared to the remaining 3 nodes. This = might further point to the possibility that the keys are hashing minimally = to the token ranges on those nodes. I'm no expert at cryptography, but = is it possible that web URLs are not evenly distributed via MD5 hashing due= to the common prefixes they contain? (such as the "http://" pref= ix, or perhaps a domain name?)? What's also interesting is that the dis= tribution is more-or-less even across alternating nodes... (0= , 2, 4 -- vs -- 1, 3).

Thanks,
Safdar


On Sun, Jun 24, 2012 at 6:00 PM, Safdar Kureishy <safdar.kureishy= @gmail.com> wrote:
Hi,

I've searched online but was unable to find any leads for the pr= oblem below. This mailing list seemed the most appropriate place. Apologies= in advance if that isn't the case.

I'm running a 5-node Solandra cluster (Solr + Cassa= ndra). I've setup the nodes with tokens evenly distributed across th= e token space, for a 5-node cluster (as evidenced below under the "= ;effective-ownership" column of the "nodetool ring" output).= =A0My data is a set of a few million crawled web pages, crawled using Nutch= , and also indexed using the "solrindex" command available throug= h Nutch. AFAIK, the key for each document generated from the crawled data i= s the URL.

Based on the "load" values for the nodes belo= w, despite adding about 3 million web pages to this index via the HTTP Rest= API (e.g.: http://9.9.9.x:8983/solandra/index/update....), some nodes= are still "empty". Specifically, nodes 9.9.9.1 and 9.9.9.3 have = just a few kilobytes (shown in bold below) of the index, while the r= emaining 3 nodes are consistently getting hammered by all the data. If the = RandomPartioner (which is what I'm using for this cluster) is supposed = to achieve an even distribution of keys across the token space, why is it t= hat the data below is skewed in this fashion? Literally, no key was yet bee= n hashed to the nodes 9.9.9.1 and 9.9.9.3 below. Could someone possibly she= d some light on this absurdity?.

[me@hm1 solandra-app]$ bin/nodetool -h hm1 ring
Address =A0 =A0 =A0 =A0 DC =A0 =A0 =A0 =A0 =A0Rack =A0 =A0 =A0 =A0Status= State =A0 Load =A0 =A0 =A0 =A0 =A0 =A0Effective-Owership =A0Token
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0136112946768375385385349842972707284580=
9.9.9.0 =A0 =A0 =A0 datacenter1 rack1 =A0 =A0 =A0 Up =A0 =A0 Normal = =A07.57 GB =A0 =A0 =A0 =A0 20.00% =A0 =A0 =A0 =A0 =A0 =A0 =A00
9.= 9.9.1 =A0 =A0 =A0 datacenter1 rack1 =A0 =A0 =A0 Up =A0 =A0 Normal =A021.= 44 KB =A0 =A0 =A0 =A020.00% =A0 =A0 =A0 =A0 =A0 =A0 =A03402823669209384= 6346337460743176821145
9.9.9.2 =A0 =A0 =A0 datacenter1 rack1 =A0 =A0 =A0 Up =A0 =A0 Normal = =A014.99 GB =A0 =A0 =A0 =A020.00% =A0 =A0 =A0 =A0 =A0 =A0 =A068056473384187= 692692674921486353642290
9.9.9.3 =A0 =A0 =A0 datacenter1 rack1 = =A0 =A0 =A0 Up =A0 =A0 Normal =A050.79 KB =A0 =A0 =A0 =A020.00% =A0 = =A0 =A0 =A0 =A0 =A0 =A0102084710076281539039012382229530463435
9.9.9.4 =A0 =A0 =A0 datacenter1 rack1 =A0 =A0 =A0 Up =A0 =A0 Normal = =A015.22 GB =A0 =A0 =A0 =A020.00% =A0 =A0 =A0 =A0 =A0 =A0 =A013611294676837= 5385385349842972707284580

Thanks in advance.
=

Regards,
Safdar

--0016e6dab06446fc2304c33b484e--