Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=ByYXvcHEDsTbSJmzfxGQqrGOKxGBKYLO8uf6M3yXq+j
	WMSNR8DChY5BSOzMaPTE0a5yWRodltHGJwTUGlpY+5ybv0SP1hBtcpFWECEWMk4g
	zr/r6RaSwVVmFUJRSkWd7tfcdTjfjO/DImSbw5OB0MTdnawm8Dh3kU6UJxqJAu3g
	=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: Cassandra to store 1 billion small 64KB Blobs
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: <AANLkTin-hBM6W___GsSBjkg2e4P0G3XTDO_AZeOtd1T9@mail.gmail.com>
Date: Tue, 27 Jul 2010 22:31:05 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <E568E5BB-CA8D-461F-84FB-D8A3604624A3@thelastpickle.com>
References: <AANLkTinHg6E_eEcGM_JVLYbBzhGoT7Ctx4_bPMEr4s5B@mail.gmail.com>
 <79329e37-3f1e-4946-a807-81cdd86655f2@me.com>
 <AANLkTi=k9kXUJXipuDk=LpLuStMt6zTzr09KQ5hBDUfb@mail.gmail.com>
 <795F2400-AB33-403A-A43B-D62DBEE59AB3@thelastpickle.com>
 <AANLkTi=mOViKCSPptg0nQ1ApPCE0Ay9e6JBpU2ex0Q0E@mail.gmail.com>
 <AANLkTin-hBM6W___GsSBjkg2e4P0G3XTDO_AZeOtd1T9@mail.gmail.com>
To: user@cassandra.apache.org

> Some possibilities open up when using OPP, especially with aggregate
> keys. This is more of an option when RF=3D=3Dcluster size, but not
> necessarily a good reason to make RF=3Dcluster size if you haven't
> already.

This use of the OOP sounds like the way Lucandra stores data, they=20
want to have range scans and some random key distribution.=20

http://github.com/tjake/Lucandra

See the hash_key() function in CassandraUtils.java for how they manually =
hash=20
the key before storing it in cassandra.=20


> 64MB per row, 1MB columns
> customerABC:file123:00000000 (colnames: 00000000, 00100000, 00200000, =
...)
> customerABC:file123:04000000 (colnames: 04000000, 04100000, ... )
> if 0xFFFFFFFF is not enough for the file size (4,294,967,295), then
> you can start with 10 or 12 digits instead (up to 2.8e+14)

Grouping together chunks into larger groups/extents is an interesting =
idea.=20
You could have a 'read ahead' buffer.  I'm=20
sure somewhere in all these designs there is a magical balance between =
row size and=20
the number of rows. They were saying chunks with the same has should =
only=20
be stored once though, so not sure if it's applicable in this case.=20

> If you needed to add metadata to chunk groups/chunks, you can use
> column names which are disjoint from '0'-'F', as long as your API
> knows how to set your predicates up likewise. If there is at least one
> column name which is dependable in each chunk row, then you can use it
> as your predicate for "what's out there" queries. This avoids loading
> column data for the chunks when looking up names (row/file/... names).
> On the other hand, if you use an empty predicate, there is not an easy
> way to avoid tombstone rows unless you make another trip to Cassandra
> to verify.

I've experimented with name spacing columns before, and found easier to=20=

use a super CF in the long run.

Cheers
Aaron
 =20