Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8291B8B1B for ; Thu, 18 Aug 2011 22:17:59 +0000 (UTC) Received: (qmail 18679 invoked by uid 500); 18 Aug 2011 22:17:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 18612 invoked by uid 500); 18 Aug 2011 22:17:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 18604 invoked by uid 99); 18 Aug 2011 22:17:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Aug 2011 22:17:55 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a49.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Aug 2011 22:17:50 +0000 Received: from homiemail-a49.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a49.g.dreamhost.com (Postfix) with ESMTP id 664025E0058 for ; Thu, 18 Aug 2011 15:17:29 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=cQIWjBJtyc ccDIdOVWxJ3t1/wZKtJN16CgnUNk6P3g81C1VuhzMALdV0TgCnsyEgwhv7CLpmWJ Ril5DS5gm+Su3vx9cTlWbDpoZHDKKkCfL2p/acruoJtAo+KLa4jWZWi6IeRf3f+j Ci9BsmpZJJ7baw9WPlYkZgQ9H56sfnMvU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=C3eBg4nW5vEocA/+ OCeZgIoUu38=; b=ecUmTLetgnMOdDuYk7XsVVewpNDGNgsosUBOWM2NkqRchZaO AREkOjCAQCnZCwd9yUXmybfWSUL877LDX5A9CzLMdFCUw3GIQqKsApxcHJWSbIuS IxdbJAZJe8Ha2x7fJ7VO/XPUl9m27srNZ2sV6IjySCKv9ZQzA/v55tqmPds= Received: from [172.16.1.2] (219-89-2-67.dialup.xtra.co.nz [219.89.2.67]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a49.g.dreamhost.com (Postfix) with ESMTPSA id 69FE85E0056 for ; Thu, 18 Aug 2011 15:17:28 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_D42D9E32-17FE-490E-B7D8-F7021AC6DE1D" Subject: Re: A few questions on row caching and read consistency ONE Date: Fri, 19 Aug 2011 10:17:25 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <95C9EE2F-53A6-4C45-80A5-DAA801336B17@thelastpickle.com> X-Mailer: Apple Mail (2.1244.3) --Apple-Mail=_D42D9E32-17FE-490E-B7D8-F7021AC6DE1D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Those numbers sound achievable. As with any scaling start with the = default config and see how you go, a 5ms response time is certainly = reasonable as are the throughput numbers. =20 e.g. If you started with 6 nodes, rf 3, with read repair turned on 20k ops -> 12k reads and 8k writes=20 X3 because of Read Repair > 36K reads , 24K writes for the whole cluster=20= per node 6k reads, 5k writes=20 per node read latency must be below 0.00016 secs per node write latency must be below 0.0002 secs 0.2 ms write latency is fine, 0.16 ms read latency may need some = attention. But you can scale up from there, and you always want to have = enough capacity to handle down nodes etc.=20 It would may be interesting to understand how many of you 2 billion keys = are hot to get a better understanding of the cache needs.=20 When you are going through the dev cycle take a look at nodetool = cfhistograms . Rows that receive writes over a long time period can = become fragmented and have a longer read path. Background = http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/ Hope that helps.=20 =20 ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 19/08/2011, at 3:27 AM, Edward Capriolo wrote: >=20 >=20 > On Thu, Aug 18, 2011 at 10:36 AM, Stephen Henderson = wrote: > Thanks Ed/Aaron, that really helped a lot. >=20 > =20 > Just to clarify on the question of writes (sorry, I worded that badly) = - do write operations insert rows into the cache on all nodes in the = replica set or does the cache only get populated on reads? >=20 > =20 > Aaron =96 in terms of scale, our ultimate goal is to achieve 99% reads = under 5ms (ideally <1ms) with upto 20,000 operations a second (split = 60/40 read/write) and upto 2 billion keys. That=92s the 12-18 month plan = at least, short-term we=92ll be more like 1000 ops/sec and 10 million = keys which I think cassandra could cope with comfortably. We=92re = currently working out what the row-size will be, but hoping to be under = 2kb max. Consistency isn=92t massively important. Our use case is as a = user-profile store for serving optimised advert-content with quite tight = restrictions on response time, so we have say 10ms to gather as much = data about a user as possible before we have to make a decision on which = creative to serve. If we can read a profile from the store in this time = we can serve a personalised ad with a higher chance of engagement so = low-latency is key requirement. >=20 > =20 > Edward =96 thanks for the link to the presentation slides. A bit = off-topic, but have you ever had a look at CouchBase (previously = =93membase=94)? It=92s basically memcached with persistence, = fault-tolerance and online scaling. It=92s the main alternative platform = we=92re considering for this project and on paper it sounds perfect, = though we have a few concerns about it (mainly lack of active community, = another nosql platform to learn and general uncertainty over the = upcoming 2.0 release). We=92re hoping to do some stress test comparison = tests between the two in the near future and I=92ll try to post the = results if they=92re not too company-specific. >=20 > =20 > Thanks again, >=20 > Stephen >=20 > =20 > From: Edward Capriolo [mailto:edlinuxguru@gmail.com]=20 > Sent: 18 August 2011 14:14 > To: user@cassandra.apache.org > Subject: Re: A few questions on row caching and read consistency ONE >=20 > =20 > =20 > On Thu, Aug 18, 2011 at 5:01 AM, Stephen Henderson = wrote: >=20 > Hi, >=20 > We're currently in the planning stage of a new project which needs a = low latency, persistent key/value store with a roughly 60:40 read/write = split. We're trying to establish if Cassandra is a good fit for this and = in particular what the hardware requirements would be to have the = majority of rows cached in memory (other nosql platforms like = Couchbase/Membase seem like a more natural fit but we're already = reasonably familiar with cassandra and would rather stick with what we = know if it can work). >=20 > If anyone could help answer/clarify the following questions it would = be a great help (all assume that row-caching is enabled for the column = family). >=20 > Q. If we're using read consistency ONE does the read request get sent = to all nodes in the replica set and the first to reply is returned (i.e. = all replica nodes will then have that row in their cache), OR does the = request only get sent to a single node in the replica set? If it's the = latter would the same node generally be used for all requests to the = same key or would it always be a random node in the replica set? (i.e. = if we have multiple reads for one key in quick succession would this = entail potentially multiple disk lookups until all nodes in the set have = been hit?). >=20 > Q. Related to the above, if only one node recieves the request would = the client (hector in this case) know which node to send the request to = directly or would there be potentially one extra network hop involved = (client -> random node -> node with key). >=20 > Q. Is it possible to do a warm cache load of the most recently = accessed keys on node startup or would we have to do this with a client = app? >=20 > Q. With write consistency ANY is it correct that following a write = request all nodes in the replica set will end up with that row in their = cache, as well as on disk, once they receive the write? i.e. total cache = size is (cache_memory_per_node * num_nodes) / num_replicas. >=20 > Q. If the cluster only has a single column family, random partitioning = and no secondary indexes, is there a good metric for estimating how much = heap space we would need to leave aside for everything that isn't the = row-cache? Would it be proportional to the row-cache size or fairly = constant? >=20 >=20 > Thanks, > Stephen >=20 >=20 > Stephen Henderson - Lead Developer (Onsite), Cognitive Match > stephen.henderson@cognitivematch.com | http://www.cognitivematch.com >=20 > =20 > I did a small presentation on this topic a while back. = http://www.edwardcapriolo.com/roller/edwardcapriolo/resource/memcache.odp >=20 >=20 > 1. >=20 > a) All reads go to all replica nodes. Even those at READ.ONE. UNLESS = you lower the read_repair_chance for the column family.=20 >=20 > b) Read could hit random nodes same node unless you confirgure dynamic = snitch to pin the request to a single node. This is described in the = cassandra.yaml >=20 > =20 > 2. Hector and no client that I know of routes requests to proper nodes = based on topology. No information of know of has proven this matters. >=20 > =20 > 3. Cassandra allows you to save your caches so your node will start up = warn (saving large rowcache is hard, large key cache is easy) >=20 > =20 > 4. Write.ANY would not change how caching works. >=20 > =20 > 5. There are some calculations out there based on size of rows. One of = the newer features of cassandra is it automatically resizes the row = cache under memory pressure now. You still have to feel it out but you = do not have to worry about setting it too high as much anymore. >=20 > =20 > One more note. I you have mentioned the row cache which is awesome it = you can utilize it correctly and your use case is prefect but key cache = + page cache can server very fast as well. >=20 > =20 > Thank you, >=20 > Edward >=20 > =20 > =20 >=20 >=20 > Wait membase is couchbase? I thought it was northscale? (I can not = keep up). It seems to have coordinators or masters. = http://www.slideshare.net/tim.lossen.de/an-introduction-to-membase=20 > Any solution where all the read write traffic travels through a master = I do not believe to be scalable. Other solutions that use a master for = coordination election but read or write directly to nodes are "more" = scalable but more fragile.=20 >=20 > Q. Why does every scalable architecture except Cassandra seem to have = master nodes ? :) >=20 > It is not in the YCSB so hard to say how fast it is or how well it = performs.=20 --Apple-Mail=_D42D9E32-17FE-490E-B7D8-F7021AC6DE1D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Those = numbers sound achievable. As with any scaling start with the default = config and see how you go, a 5ms response time is certainly reasonable = as are the throughput numbers.  

e.g. If you = started with 6 nodes, rf 3, with read repair turned = on

20k ops -> 12k reads and 8k = writes 
X3 because of Read Repair > 36K reads , 24K = writes for the whole cluster 
per node 6k reads, 5k = writes 
per node read latency must be below 0.00016 = secs
per node write latency must be below 0.0002 = secs

0.2 ms write latency is fine, 0.16 ms read = latency may need some attention. But you can scale up from there, = and you always want to have enough capacity to handle down nodes = etc. 

It would may be interesting to = understand how many of you 2 billion keys are hot to get a better = understanding of the cache needs. 

When you are = going through the dev cycle take a look at nodetool cfhistograms . Rows = that receive writes over a long time period can become fragmented and = have a longer read path. Background htt= p://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
<= br>
Hope that helps. 
 
http://www.thelastpickle.com

On 19/08/2011, at 3:27 AM, Edward Capriolo wrote:



On Thu, Aug 18, 2011 at 10:36 AM, Stephen = Henderson <stephen.henderson@cog= nitivematch.com> wrote:

Thanks = Ed/Aaron, that really helped a lot.

 

Just to clarify on the question = of writes (sorry, I worded that badly) - do write operations insert rows = into the cache on all nodes in the replica set or does the cache only = get populated on reads?

 

Aaron =96 in terms of scale, = our ultimate goal is to achieve 99% reads under 5ms (ideally <1ms) = with upto 20,000 operations a second (split 60/40 read/write) and upto 2 = billion keys. That=92s the 12-18 month plan at least, short-term we=92ll = be more like 1000 ops/sec and 10 million keys which I think cassandra = could cope with comfortably. We=92re currently working out what the = row-size will be, but hoping to be under 2kb max.  Consistency = isn=92t massively important. Our use case is as a user-profile store for = serving optimised advert-content with quite tight restrictions on = response time, so we have say 10ms to gather as much data about a user = as possible before we have to make a decision on which creative to = serve. If we can read a profile from the store in this time we can serve = a personalised ad with a higher chance of engagement so low-latency is = key requirement.

 

Edward =96 thanks for the link = to the presentation slides. A bit off-topic, but have you ever had a = look at CouchBase (previously =93membase=94)? It=92s basically memcached = with persistence, fault-tolerance and online scaling. It=92s the main = alternative platform we=92re considering for this project and on paper = it sounds perfect, though we have a few concerns about it (mainly lack = of active community, another nosql platform to learn and general = uncertainty over the upcoming 2.0 release). We=92re hoping to do some = stress test comparison tests between the two in the near future and I=92ll= try to post the results if they=92re not too = company-specific.

 

Thanks again,

Stephen

 

From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: 18 August 2011 14:14
To: user@cassandra.apache.org
Subject: Re: A = few questions on row caching and read consistency = ONE

 
 

On = Thu, Aug 18, 2011 at 5:01 AM, Stephen Henderson <stephen.henderson@cognitivematch.com> = wrote:

Hi,

We're currently in the = planning stage of a new project which needs a low latency, persistent = key/value store with a roughly 60:40 read/write split. We're trying to = establish if Cassandra is a good fit for this and in particular what the = hardware requirements would be to have the majority of rows cached in = memory (other nosql platforms like Couchbase/Membase seem like a more = natural fit but we're already reasonably familiar with cassandra and = would rather stick with what we know if it can work).

If anyone could help answer/clarify the following questions it would = be a great help (all assume that row-caching is enabled for the column = family).

Q. If we're using read consistency ONE does the read = request get sent to all nodes in the replica set and the first to reply = is returned (i.e. all replica nodes will then have that row in their = cache), OR does the request only get sent to a single node in the = replica set? If it's the latter would the same node generally be used = for all requests to the same key or would it always be a random node in = the replica set? (i.e. if we have multiple reads for one key in quick = succession would this entail potentially multiple disk lookups until all = nodes in the set have been hit?).

Q. Related to the above, if only one node recieves the request would = the client (hector in this case) know which node to send the request to = directly or would there be potentially one extra network hop involved = (client -> random node -> node with key).

Q. Is it possible to do a warm cache load of the most recently = accessed keys on node startup or would we have to do this with a client = app?

Q. With write consistency ANY is it correct that following a = write request all nodes in the replica set will end up with that row in = their cache, as well as on disk, once they receive the write? i.e. total = cache size is (cache_memory_per_node * num_nodes) / num_replicas.

Q. If the cluster only has a single column family, random = partitioning and no secondary indexes, is there a good metric for = estimating how much heap space we would need to leave aside for = everything that isn't the row-cache? Would it be proportional to the = row-cache size or fairly constant?


Thanks,
Stephen


Stephen Henderson - Lead Developer = (Onsite), Cognitive Match
stephen.henderson@cognitivematch.com | http://www.cognitivematch.com

 

I did a small presentation on this topic a while = back. http://www.edwardcapriolo.com/roller/edwardcapriolo/reso= urce/memcache.odp


1.

a) All reads go to all replica nodes. Even those at = READ.ONE. UNLESS you lower the read_repair_chance for the column = family. 

b) Read could hit random nodes same node unless you confirgure dynamic = snitch to pin the request to a single node. This is described in the = cassandra.yaml

 

2. Hector and no client that I know of routes requests to proper nodes = based on topology. No information of know of has proven this = matters.

 

3. Cassandra allows you to save your caches so your = node will start up warn (saving large rowcache is hard, large key cache = is easy)

 

4. Write.ANY would not change how caching = works.

 

5. There are some calculations out there based on = size of rows. One of the newer features of cassandra is it automatically = resizes the row cache under memory pressure now. You still have to feel = it out but you do not have to worry about setting it too high as much = anymore.

 

One more note. I you have mentioned the row cache = which is awesome it you can utilize it correctly and your use case is = prefect but key cache + page cache can server very fast as well.

 

Thank you,

Edward

 
 


Wait membase is couchbase? I = thought it was northscale? (I can not keep up). It seems to have = coordinators or masters.  http://www.slideshare.net/tim.lossen.de/an-introduction-to-membase&n= bsp;
Any solution where all the read write traffic travels through a = master I do not believe to be scalable. Other solutions that use a = master for coordination election but read or write directly to nodes are = "more" scalable but more fragile. 

Q. Why does every scalable architecture except = Cassandra seem to have master nodes ? :)

It is = not in the YCSB so hard to say how fast it is or how well it = performs. 

= --Apple-Mail=_D42D9E32-17FE-490E-B7D8-F7021AC6DE1D--