Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E7B999760 for ; Sun, 24 Jun 2012 17:39:38 +0000 (UTC) Received: (qmail 61831 invoked by uid 500); 24 Jun 2012 17:39:36 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 61804 invoked by uid 500); 24 Jun 2012 17:39:36 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61794 invoked by uid 99); 24 Jun 2012 17:39:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jun 2012 17:39:36 +0000 X-ASF-Spam-Status: No, hits=4.0 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,TRACKER_ID,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [72.35.23.45] (HELO smtp-out2.electric.net) (72.35.23.45) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jun 2012 17:39:30 +0000 Received: from [10.86.10.81] (helo=fuseout2a.electric.net) by anders.electric.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.77) (envelope-from ) id 1SiqmC-0004X9-Vv for user@cassandra.apache.org; Sun, 24 Jun 2012 10:39:08 -0700 Received: from mailanyone.net by fuseout2a.electric.net with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (MailAnyone extSMTP dbrosius@baybroadband.net) id 1SiqmC-0006eD-2o for user@cassandra.apache.org; Sun, 24 Jun 2012 10:39:08 -0700 Message-ID: <4FE75097.7040204@mebigfatguy.com> Date: Sun, 24 Jun 2012 13:38:31 -0400 From: Dave Brosius User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20120430 Thunderbird/12.0.1 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: Re: RandomPartitioner is providing a very skewed distribution of keys across a 5-node Solandra cluster References: In-Reply-To: Content-Type: multipart/alternative; boundary="------------090307000707070200080501" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------090307000707070200080501 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit If i read what you are saying, you are _not_ using composite keys? That's one thing that could do it, if the first part of the composite key had a very very low cardinality. On 06/24/2012 11:00 AM, Safdar Kureishy wrote: > Hi, > > I've searched online but was unable to find any leads for the problem > below. This mailing list seemed the most appropriate place. Apologies > in advance if that isn't the case. > > I'm running a 5-node Solandra cluster (Solr + Cassandra). I've setup > the nodes with tokens /evenly distributed across the token space/, for > a 5-node cluster (as evidenced below under the "effective-ownership" > column of the "nodetool ring" output). My data is a set of a few > million crawled web pages, crawled using Nutch, and also indexed using > the "solrindex" command available through Nutch. AFAIK, the key for > each document generated from the crawled data is the URL. > > Based on the "load" values for the nodes below, despite adding about 3 > million web pages to this index via the HTTP Rest API (e.g.: > http://9.9.9.x:8983/solandra/index/update....), some nodes are still > "empty". Specifically, nodes 9.9.9.1 and 9.9.9.3 have just a few > kilobytes (shown in *bold* below) of the index, while the remaining 3 > nodes are consistently getting hammered by all the data. If the > RandomPartioner (which is what I'm using for this cluster) is supposed > to achieve an even distribution of keys across the token space, why is > it that the data below is skewed in this fashion? Literally, no key > was yet been hashed to the nodes 9.9.9.1 and 9.9.9.3 below. Could > someone possibly shed some light on this absurdity?. > > [me@hm1 solandra-app]$ bin/nodetool -h hm1 ring > Address DC Rack Status State Load > Effective-Owership Token > > 136112946768375385385349842972707284580 > 9.9.9.0 datacenter1 rack1 Up Normal 7.57 GB > 20.00% 0 > 9.9.9.1 datacenter1 rack1 Up Normal *21.44 KB* > 20.00% 34028236692093846346337460743176821145 > 9.9.9.2 datacenter1 rack1 Up Normal 14.99 GB > 20.00% 68056473384187692692674921486353642290 > 9.9.9.3 datacenter1 rack1 Up Normal *50.79 KB* > 20.00% 102084710076281539039012382229530463435 > 9.9.9.4 datacenter1 rack1 Up Normal 15.22 GB > 20.00% 136112946768375385385349842972707284580 > > Thanks in advance. > > Regards, > Safdar --------------090307000707070200080501 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit If i read what you are saying, you are _not_ using composite keys? That's one thing that could do it, if the first part of the composite key had a very very low cardinality.

On 06/24/2012 11:00 AM, Safdar Kureishy wrote:
Hi,

I've searched online but was unable to find any leads for the problem below. This mailing list seemed the most appropriate place. Apologies in advance if that isn't the case.

I'm running a 5-node Solandra cluster (Solr + Cassandra). I've setup the nodes with tokens evenly distributed across the token space, for a 5-node cluster (as evidenced below under the "effective-ownership" column of the "nodetool ring" output). My data is a set of a few million crawled web pages, crawled using Nutch, and also indexed using the "solrindex" command available through Nutch. AFAIK, the key for each document generated from the crawled data is the URL.

Based on the "load" values for the nodes below, despite adding about 3 million web pages to this index via the HTTP Rest API (e.g.: http://9.9.9.x:8983/solandra/index/update....), some nodes are still "empty". Specifically, nodes 9.9.9.1 and 9.9.9.3 have just a few kilobytes (shown in bold below) of the index, while the remaining 3 nodes are consistently getting hammered by all the data. If the RandomPartioner (which is what I'm using for this cluster) is supposed to achieve an even distribution of keys across the token space, why is it that the data below is skewed in this fashion? Literally, no key was yet been hashed to the nodes 9.9.9.1 and 9.9.9.3 below. Could someone possibly shed some light on this absurdity?.

[me@hm1 solandra-app]$ bin/nodetool -h hm1 ring
Address         DC          Rack        Status State   Load            Effective-Owership  Token
                                                                                           136112946768375385385349842972707284580
9.9.9.0       datacenter1 rack1       Up     Normal  7.57 GB         20.00%              0
9.9.9.1       datacenter1 rack1       Up     Normal  21.44 KB        20.00%              34028236692093846346337460743176821145
9.9.9.2       datacenter1 rack1       Up     Normal  14.99 GB        20.00%              68056473384187692692674921486353642290
9.9.9.3       datacenter1 rack1       Up     Normal  50.79 KB        20.00%              102084710076281539039012382229530463435
9.9.9.4       datacenter1 rack1       Up     Normal  15.22 GB        20.00%              136112946768375385385349842972707284580

Thanks in advance.

Regards,
Safdar

--------------090307000707070200080501--