Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of peter78@gmail.com designates
 209.85.222.181 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:from:in-reply-to:mime-version:date:message-id:subject:to
         :content-type;
        b=sz6DWEHmGPnW5JzkohSjDucUexrZKZ2m1Eu8qYuRj/ISTq3l7XuQisNKm333ghiSEv
         NgDX7VnAJty8vF4u23UozQ4/4FAtecaNMQs1XCLlO4fwy/K/Hfc519mQ7Un2ZVa12V+Z
         aU+XJJkpkEh9dJLMbC8LPCZbwEVbBuVs9SPF4=
References: <i2y7e99ee5f1004152328x8fb29b12jc18e8e4dd38e995@mail.gmail.com>  
	<p2r7ca6fc441004160138lbed0a77ch35e28f0288b30368@mail.gmail.com>  
	<4BC88D97.7050104@gmail.com> <-6358312770452844291@unknownmsgid>
	<y2q828083e71004161102ze1f9d8eav9fbf9b61426cfe22@mail.gmail.com>
From: Peter Chang <peter78@gmail.com>
In-Reply-To: <y2q828083e71004161102ze1f9d8eav9fbf9b61426cfe22@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 7E18)
Date: Fri, 16 Apr 2010 14:42:18 -0600
Message-ID: <-8331721885605240603@unknownmsgid>
Subject: Re: Regarding Cassandra Scalability
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Yeah. I wasn't sure if Cassandra was optimized for binary data
especially since any site of that size will use a CDN. Interesting
read though.

I think 1K per tweet is off by an order of magnitude considering they
only allow 140 characters. Regardless the number of users with > 1MM
is probably a handful. Also im guessing they purge data after a
certain window (like 30 days for example).

Sent from my iPhone


On Apr 16, 2010, at 12:02 PM, gabriele renzi <rff.rff@gmail.com> wrote:

> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang <peter78@gmail.com>
> wrote:
>> FB also does pics and movies so 1MB is way off depending on where
>> they
>> manage such binary data.
>
> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>
>> I do agree that 1MB of text alone is a lot of text
>> which is more relevant in the case of Twitter. The only large thing
>> you
>> leave out is denormalization. Every tweet you write is likely
>> denormalized
>> across your followers to allow for quick read access.
>
> .. but considering many users have _millions_ of followers, this may
> be quite a bit more data. Assuming 1k per tweet, this would mean one
> from @aplusk (4.7M followers) would take more than 4 gigabytes of
> data. Assuming ten tweets a day, in one month he'd produce one TB.
>
> I'd say they only store references (increasing number lists can also
> be encoded very cleverly), or in some other way I'm not smart enough
> to think of.