incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Schema recommendation
Date Mon, 07 Jan 2013 03:59:04 GMT
> - basically the task is to create a distributed file system with only few allowed operations
Background http://www.datastax.com/dev/blog/cassandra-file-system-design

> - it should handle split brain conditions well (two or three DCs, it's possible to get
user requests while the intra-DC connections are down)
Handling entity creation in disconnected DC's can be a little tricky. if it's possible for
multiple entities with the same "natural key" / "path" to be created you can either have last
write wins or handle it at the application level. To do that you need to find a way for code
in each DC to write to a different place / column. Then resolve the differences in application
at read time. 

> All (minus the last) should be close to ACID.
Nope. 
We have Atomicity, Isolation and Durability. Not going to get Consistency in the ACID sense.
See the first presentation here http://www.datastax.com/events/cassandrasummit2012/presentations

> This brought me the following schema:
Take a look at the one linked at the top. 
You probably want to disconnect the file contents from the directory listing. And if listing
directory contents is a hot code path I would consider supporting that by having a directory
CF. 
 
> I could add more CF(s) for example with composite columns (each directory being a row),
but that would destroy the benefit of the above schema for moving files in one (atomic and
idempotent) operation. 
If you store the directory listing differently from the file moving it is a (essentially)
meta data operation. 
Any write is atomic in the sense that either all changes commit or none do. And they writes
other than counter increments are idempotent. 

For what it's worth, Riak has an out of the box S3 compatible service http://basho.com/products/riakcs/.
Not sure what the pricing is. 

Hope that helps. 

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 7/01/2013, at 4:46 AM, Attila Nagy <bra@fsn.hu> wrote:

> Hi,
> 
> I'm new to Cassandra, so I wonder what would be the most efficient(*) schema for specific
operations on the following dataset:
> - basically the task is to create a distributed file system with only few allowed operations
> - it should handle split brain conditions well (two or three DCs, it's possible to get
user requests while the intra-DC connections are down)
> - each data file is blob (pretty often, but not always 8 bit text, sometimes without
the encoding known), ranging from a few 100 bytes to around 20-30 MB. Average size is 350
kB.
> - it's well compressible (gzip -1 gives 1.63x compress ratio on plain files)
> - files are mostly immutable
> - for the few, which are not, they are append-only
> - each file has a unique file name (size in the range of 70-100 bytes) and it's preserved
during its lifetime (if it changes, a rewrite is OK).
> - the files have metadata attached (various, 1.2's sets and maps are a good fit here,
but even simple columns should do)
> - the files are organized into directories (multi level, and sometimes there can be up
to some millions of files in a dir, but more likely are the range of 0-some hundred, thousands
(up to 10k))
> - directories also have metadata (most notably an mtime, which changes when directory
contents are changed, that can be used to cache directory lists)
> - each directory (and the files therein) belongs to a user(name)
> - Cassandra 1.2 is fine
> 
> Given that designing schema for Cassandra begins with listing the operations, here they
are:
> 1. get the contents of a directory (input: directory (owner) name, output:file names)
> 2. get a file (in: file (dir) name, out:file metadata and contents)
> 3. put a file (in: file (dir, owner) name, metadata, data)
> 4. append to a file, chunk size maximum is 4-8 kB (in: file (dir, owner) name, data)
> 5. move a file to a different directory (in: file, dir (owner) name, target dir)
> 6. remove a file (in: file (dir, owner) name)
> 7. remove all stuff for a user (input: user name), but it's "rare" (compared to the above),
so walking through the dirlist on the client side is OK, it's not performance critical
> 
> All (minus the last) should be close to ACID.
> 
> I've tried to do the homework (taken a look at Cassandra in the 0.6-7 times, so now I'm
le-learning the new (1.2, CQL 3) way) and still couldn't find the best way.
> 
> I thought the best would be to start with the docs, without any preliminary performance
testing.
> 
> This brought me the following schema:
> CREATE TABLE file (
>   name varchar PRIMARY KEY,
>   owner varchar,
>   dir varchar,
>   flags set<ascii>,
>   fstat map<ascii, int>,
>   data list<blob>
> );
> CREATE INDEX file_dir ON file (dir);
> CREATE INDEX file_owner ON file (owner);
> 
> Which gives me for the operations:
> 1. something like SELECT name(,etc) FROM file WHERE dir="dirname"; Which can be LIMIT-ed.
Problems: SLOOOW (and maybe despite the LIMIT, it materializes in the coordinator's memory,
I don't know), also, doesn't scale, because all nodes must inspect their index CFs.
> 2. a simple SELECT data(,etc) FROM file WHERE name="filename"; Problems: Cassandra is
said not to good at storing such amounts of data, it has to read all in memory (on the coordinator
and the replica node), also the client will have to hold it in memory. But it seems to be
acceptible to some levels, because all data is     needed, so fetching it once is an optimization.
The limiting factor here seems to be the network speed (nodes pass the data as a hot potato,
the slower the network, the longer it has to be kept in memory), and CPU speed.
> 3. a simple INSERT INTO or UPDATE
> 4. 1.2's lists make appends easy
> 5. most important. It's a manner of UPDATE file SET dir VALUES("newdir") WHERE name="filename";
This either happens, or not, there is no situation where the file is in multiple directories,
or nowhere. Even if a site (node) doesn't get it, it still sees the file on the old location,
and if there is a move there, eventually everything will get into the right shape, without
worrying needed on the client side.
> 6. a simple DELETE
> 7. SELECT and DELETEs
> 
> For most operations, it seems fine (but your insightful recommendations are welcome :),
the biggest pain seems to be listing directories.
> It takes seconds from minutes (or timeouts).
> I could add more CF(s) for example with composite columns (each directory being a row),
but that would destroy the benefit of the above schema for moving files in one (atomic and
idempotent) operation. 
> 
> Is there a schema, which can do best for all operations and still maintain ACID-like
properties?
> 
> Thanks,
> 
> *: in terms of query efficiency, closeness to ACID


Mime
View raw message