Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 54701 invoked from network); 9 Jul 2010 17:48:40 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 9 Jul 2010 17:48:40 -0000 Received: (qmail 3128 invoked by uid 500); 9 Jul 2010 17:48:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 3045 invoked by uid 500); 9 Jul 2010 17:48:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 3035 invoked by uid 99); 9 Jul 2010 17:48:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 17:48:39 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.160.44] (HELO mail-pw0-f44.google.com) (209.85.160.44) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 17:48:31 +0000 Received: by pwj1 with SMTP id 1so1160089pwj.31 for ; Fri, 09 Jul 2010 10:48:09 -0700 (PDT) Received: by 10.142.158.12 with SMTP id g12mr11864050wfe.186.1278697286681; Fri, 09 Jul 2010 10:41:26 -0700 (PDT) Received: from [10.0.1.4] (173-164-32-245-colorado.hfc.comcastbusiness.net [173.164.32.245]) by mx.google.com with ESMTPS id r27sm967596rvq.9.2010.07.09.10.41.23 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 09 Jul 2010 10:41:25 -0700 (PDT) From: Joe Stump Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: multipart/alternative; boundary=Apple-Mail-197--257628492 Subject: Re: RackAwareStrategy vs RackUnAwareStrategy on AWS EC2 cloud Date: Fri, 9 Jul 2010 11:41:22 -0600 In-Reply-To: <511110.57261.qm@web112013.mail.gq1.yahoo.com> To: user@cassandra.apache.org References: <511110.57261.qm@web112013.mail.gq1.yahoo.com> Message-Id: <08CD64CB-33B0-4106-A703-E392FC04C374@joestump.net> X-Mailer: Apple Mail (2.1081) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-197--257628492 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii We had similar issues when we started running Cassandra on EC2 between = multiple AZ's (not regions; we're working up to that shortly). We ended = up building a rack aware strategy specific to AWS, which is posted = somewhere in JIRA. Basically it uses the AWS API to ensure that = replicants are stored in each AZ. We then ensure that our clients are = only reading from nodes in a given AZ. What I'm guessing is that you're = seeing latency issues between regions combined with a higher consistency = level than what we use. Also, SSL tunnels are hard to scale from the management side. The Amazon = folks have told us that VPNCubed is a better solution for such things. --Joe On Jul 9, 2010, at 11:36 AM, maneela a wrote: > Are there any known performance issues if cassandra cluster launched = with RackAwareStrategy because I see huge performance difference between = RackAwareStrategy vs RackUnAwareStrategy. Here are details: >=20 >=20 > we have a cluster setup with 4 EC2 X large nodes, 3 of them are = running in East region and 4th one is running in West region and they = all communicate with each other through VPN tunnel interface which is = only way we found to achieve ring architecture across Amazon cloud = regions: >=20 >=20 > we are able to process 3.5K write operations per second when we used = RackUnAwareStrategy whereas=20 >=20 > :/home/ubuntu/cassandra/contrib/py_stress# ./stress.py -o insert -n = 80000 -y regular -d ec2-xxx-xxx-xxx-xx.compute-1.amazonaws.com --threads = 100 --keep-going > total,interval_op_rate,avg_latency,elapsed_time > 35935,3593,0.0289930914479,10 > 70531,3459,0.0289145907593,20 > 80000,946,0.0267288666213,30 >=20 > whereas we are able to process only 250 write operations per second = when we used RackAwareStrategy >=20 > :/home/ubuntu/cassandra/contrib/py_stress# ./stress.py -o insert -n = 80000 -y regular -d ec2-xxx-xxx-xxx-xx.compute-1.amazonaws.com --threads = 100 --keep-going > total,interval_op_rate,avg_latency,elapsed_time > 2327,232,0.434396038355,10 > 4772,244,0.40946514036,20 > 7383,261,0.384504625415,30 > 9924,254,0.392919449861,40 > 12525,260,0.383832110482,50 > 15158,263,0.378838069983,60 > 17784,262,0.383219807364,70 > 20416,263,0.381646275973,80 > 23030,261,0.382550528602,90 > 25644,261,0.384442176815,100 > 28268,262,0.380935921084,110 > 30910,264,0.377376309224,120 > 33541,263,0.385158945698,130 > 36119,257,0.387976026517,140 > 38735,261,0.382333525368,150 > 41342,260,0.38413751514,160 > 43925,258,0.387684800391,170 > 46642,271,0.36899637237,180 > 49291,264,0.378489510164,190 > 51931,264,0.3793784538,200 > 54573,264,0.378474057217,210 > 57253,268,0.374258003573,220 > 59884,263,0.380020038658,230 > 62484,260,0.387267011954,240 > 64728,224,0.439328571054,250 > 67340,261,0.389221810455,260 > 69920,258,0.386144905127,270 > 72531,261,0.384242234948,280 > 75202,267,0.372129596605,290 > 77843,264,0.354621512291,300 > 80000,215,0.183918378283,310 >=20 > Thanks in advance >=20 > Niru >=20 --Apple-Mail-197--257628492 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Are there any known performance issues if = cassandra cluster launched with RackAwareStrategy because I see huge = performance difference between RackAwareStrategy vs = RackUnAwareStrategy.  Here are details:


we = have a cluster setup with 4 EC2 X large nodes, 3 of them are running in = East region and 4th one is running in West region and they all = communicate with each other through VPN tunnel interface which is only = way we found to achieve ring architecture across Amazon cloud = regions:


we are able to process 3.5K write = operations per second when we used RackUnAwareStrategy = whereas 

ec2-xxx-xxx-xxx= -xx.compute-1.amazonaws.com --threads 100 --keep-going


ec2-xxx-xxx-xxx= -xx.compute-1.amazonaws.com --threads 100 --keep-going