cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Widmann <michael.widm...@gmail.com>
Subject Re: Cassandra to store 1 billion small 64KB Blobs
Date Mon, 26 Jul 2010 15:06:59 GMT
Okay . That really made a knot into my brain - It twist's a little bit now
I've to draw that  on the whiteboard to understand it better ... but I've
seen some very interesting cornerstones in your answer
for our project.

really thanks a lot
mike

2010/7/26 aaron morton <aaron@thelastpickle.com>

> I see, got carried away thinking about it so here are some thoughts....
>
> Your access patterns will determine the best storage design, so it's
> probably not the best solution. I would welcome thoughts from others.
>
> => Standard CF: Chunks
>
> * key is chunk hash
> * col named 'data' col value is chunk data
> * another col with name "hash" and value of the hash.
> * could also store access data against the individual chunks here
> * probably cache lots of keys, and a only a few rows
>
> - to test if the hashes you have already exist do a multi get slice . But
> you need to specify a col to return the value for, and not the big data one.
> So use the hash column.
>
> => Standard CF: CustomerFiles
>
> * key is customer id
> * col per file name, col value perhaps latest version number or last
> accessed
> * cache keys and rows
>
> - to get all the files for a customer slice the customers row
> - if you want access data when you list all files for a customer, use a
> super CF with a super col for each file name. Store the meta data for the
> file in this CF *and* in the Files CF
>
> => Super CF: Files
>
> * key is client_id.file_name
> * super column called "meta" with path / accessed etc. including versions
> * super column called "current" with columns named "0001" and col values as
> the chunk hash
> * super column called "version.X" with the same format as above for current
> * cache keys and rows
>
> - assumes meta is shared across versions
> - to rebuild a file get_slice for all cols in the "current" super col and
> then do multi gets for the chunks
> - row grows as versions of file grows, but only storing links to the chunks
> so probably OK. Consider how many versions X how many chunks, then may way
> to make the number of rows grow instead of row size. Perhaps have a
> FileVersions CF where the key includes the version number, then maintain
> information about the current version in both Files and FileVersions CFs.
> Files CF would only ever have the current file, update access meta data in
> both CF's.
>
> => Standard CF: ChunkUsage
>
> * key is chunk hash
> * col name is a versioned file key cust_id.file_name.version col value is
> the count of usage in that version
> * no caching
>
> - cannot increase a counter until cassandra 0.7, so cannot keep a count of
> chunk usage.
> - this is the reverse index to see where the chunk is used.
> - does not consider what is current, just all time usage.
> - not sure how much re-use their is, but row size grows with reuse. Should
> be ok for couple of million cols.
>
>
> Oh and if your going to use Hadoop / PIG to analyse the data in this
> beastie you need to think about that in the design. You'll probably want
> single CF to serve the queries with.
>
> Hope that helps
> Aaron
>
>
> On 26 Jul 2010, at 17:29, Michael Widmann wrote:
>
> Hi
>
> Wow that was lot of information...
>
> Think about users storing files online (means with their customer name) -
> each customer maintains his own "hashtable" of files. Each File can consist
> of some or several thousand entries (depends on the size of the whole file).
>
>
> for example:
>
> File Test.doc  consists of 3 * 64K Blobs  - each blob does have the hash
> value ABC - so we will only store one blob, one hash value and the entry
> that this blob is needed 3 times, so we try to avoid duplicate file data
> (and there's a lot of duplicates)
>
> Means our modell is:
>
> Filename:
> Test.doc
> Hashes: 3
> Hash 1: ABC
> Hash 2: ABC
> Hash 3: ABC
> BLOB:ABC
>  Used: 3
>  Binary: Data 1*
>
> Each customer does have:
>
> Customer:  Customer:ID
>
> Filename: Test.doc
>  MetaData (Path / Accessed / modified / Size / compression / OS-Type from /
> Security)
>  Version: 0
>  Hash 1 / Hash 2 / Hash 3
>
> Filename: Another.doc
>
>   MetaData (Path / Accessed / modified / Size / compression / OS-Type from
> / Security)
>   Version: 0
>   Hash 1 / Hash 2 / Hash 3
>   Version: 1
>   Hash 1 / Hash 2 / Hash 4
>   Version: 2
>   Hash 3 / Hash 2 / Hash 2
>
> Hope this clear some things :-)
>
> Mike
>
>
> 2010/7/26 Aaron Morton <aaron@thelastpickle.com>
>
>> Some background reading..
>> http://ria101.wordpress.com/2010/02/22/cassandra-randompartitioner-vs-orderpreservingpartitioner/
>>
>> Not sure on your follow up question, so I'll just wildly blather on about
>> things :)
>>
>> My assumption of your data is you have 64K chunks that are identified by a
>> hash, which can somehow be grouped together into larger files (so there is a
>> "file name" of sorts).
>>
>> One possible storage design (assuming the Random Partitioner) is....
>>
>> A Chunks CF, each row in this CF uses the hash of the chunk as it's key
>> and has is a single column with the chunk data. You could use more columns
>> to store meta here.
>>
>> A ChunkIndex CF, each row uses the file name (from above) as the key and
>> has one column for each chunk in the file. The column name *could* be an
>> offset for the chunk and the column value could be the hash for the chunk.
>> Or you could use the chunk hash as the col name and the offset as the col
>> value if needed.
>>
>> To rebuild the file read the entire row from the ChunkIndex, then make a
>> series of multi gets to read all the chunks. Or you could lazy populate the
>> ones you needed.
>>
>> This is all assuming that the 1000's comment below means you could want to
>> combine the chunks  60+ MB chunks. It would be easier to keep all the chunks
>> together in one row, if you are going to have large (unbounded) file size
>> this may not be appropriate.
>>
>> You could also think about using the order preserving partitioner, and
>> using a compound key for each row such as "file_name_hash.offset" . Then by
>> using the get_range_slices to scan the range of chunks for a file you would
>> not need to maintain a secondary index. Some drawbacks to that approach,
>> read the article above.
>>
>> Hope the helps
>> Aaron
>>
>>
>>
>> On 26 Jul, 2010,at 04:01 PM, Michael Widmann <michael.widmann@gmail.com>
>> wrote:
>>
>> Thanks for this detailed description ...
>>
>> You mentioned the secondary index in a standard column, would it be better
>> to build several indizes?
>> Is that even possible to build a index on for example 32 columns?
>>
>> The hint with the smaller boxes is very valuable!
>>
>> Mike
>>
>> 2010/7/26 Aaron Morton <aaron@thelastpickle.com>
>>
>>> For what it's worth...
>>>
>>> * Many smaller boxes with local disk storage are preferable to 2 with
>>> huge NAS storage.
>>> * To cache the hash values look at the KeysCached setting in the
>>> storage-config
>>> * There are some row size limits see
>>> http://wiki.apache.org/cassandra/CassandraLimitations
>>> * If you wanted to get 1000 blobs, rather then group them in a single row
>>> using a super column consider building a secondary index in a standard
>>> column. One CF for the blobs using your hash, one CF that uses whatever they
>>> grouping key is with a col for every blobs hash value. Read from the index
>>> first, then from the blobs themselves.
>>>
>>> Aaron
>>>
>>>
>>>
>>> On 24 Jul, 2010,at 06:51 PM, Michael Widmann <michael.widmann@gmail.com>
>>> wrote:
>>>
>>> Hi Jonathan
>>>
>>> Thanks for your very valuable input on this.
>>>
>>> I maybe didn't enough explanation - so I'll try to clarify
>>>
>>> Here are some thoughts:
>>>
>>>
>>>    - binary data will not be indexed - only stored.
>>>    - The file name to the binary data (a hash) should be indexed for
>>>    search
>>>    - We could group the hashes in 62 "entry" points for search
>>>    retrieving -> i think suprcolumns (If I'm right in terms) (a-z,A_Z,0-9)
>>>    - the 64k Blobs meta data (which one belong to which file) should be
>>>    stored separate in cassandra
>>>    - For Hardware we rely on solaris / opensolaris with ZFS in the
>>>    backend
>>>    - Write operations occur much more often than reads
>>>    - Memory should hold the hash values mainly for fast search (not the
>>>    binary data)
>>>    - Read Operations (restore from cassandra) may be async - (get about
>>>    1000 Blobs) - group them restore
>>>
>>> So my question is too:
>>>
>>> 2 or 3 Big boxes or 10 till 20 small boxes for storage...
>>> Could we separate "caching" - hash values CFs cashed and indexed - binary
>>> data CFs not ...
>>> Writes happens around the clock - on not that tremor speed but constantly
>>>
>>> Would compaction of the database need really much disk space
>>> Is it reliable on this size (more my fear)
>>>
>>> thx for thinking and answers...
>>>
>>> greetings
>>>
>>> Mike
>>>
>>> 2010/7/23 Jonathan Shook <jshook@gmail.com>
>>>
>>>> There are two scaling factors to consider here. In general the worst
>>>> case growth of operations in Cassandra is kept near to O(log2(N)). Any
>>>> worse growth would be considered a design problem, or at least a high
>>>> priority target for improvement.  This is important for considering
>>>> the load generated by very large column families, as binary search is
>>>> used when the bloom filter doesn't exclude rows from a query.
>>>> O(log2(N)) is basically the best achievable growth for this type of
>>>> data, but the bloom filter improves on it in some cases by paying a
>>>> lower cost every time.
>>>>
>>>> The other factor to be aware of is the reduction of binary search
>>>> performance for datasets which can put disk seek times into high
>>>> ranges. This is mostly a direct consideration for those installations
>>>> which will be doing lots of cold reads (not cached data) against large
>>>> sets. Disk seek times are much more limited (low) for adjacent or near
>>>> tracks, and generally much higher when tracks are sufficiently far
>>>> apart (as in a very large data set). This can compound with other
>>>> factors when session times are longer, but that is to be expected with
>>>> any system. Your storage system may have completely different
>>>> characteristics depending on caching, etc.
>>>>
>>>> The read performance is still quite high relative to other systems for
>>>> a similar data set size, but the drop-off in performance may be much
>>>> worse than expected if you are wanting it to be linear. Again, this is
>>>> not unique to Cassandra. It's just an important consideration when
>>>> dealing with extremely large sets of data, when memory is not likely
>>>> to be able to hold enough hot data for the specific application.
>>>>
>>>> As always, the real questions have lots more to do with your specific
>>>> access patterns, storage system, etc I would look at the benchmarking
>>>> info available on the lists as a good starting point.
>>>>
>>>>
>>>> On Fri, Jul 23, 2010 at 11:51 AM, Michael Widmann
>>>> <michael.widmann@gmail.com> wrote:
>>>> > Hi
>>>> >
>>>> > We plan to use cassandra as a data storage on at least 2 nodes with
>>>> RF=2
>>>> > for about 1 billion small files.
>>>> > We do have about 48TB discspace behind for each node.
>>>> >
>>>> > now my question is - is this possible with cassandra - reliable -
>>>> means
>>>> > (every blob is stored on 2 jbods)..
>>>> >
>>>> > we may grow up to nearly 40TB or more on cassandra "storage" data ...
>>>> >
>>>> > anyone out did something similar?
>>>> >
>>>> > for retrieval of the blobs we are going to index them with an
>>>> hashvalue
>>>> > (means hashes are used to store the blob) ...
>>>> > so we can search fast for the entry in the database and combine the
>>>> blobs to
>>>> > a normal file again ...
>>>> >
>>>> > thanks for answer
>>>> >
>>>> > michael
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> bayoda.com - Professional Online Backup Solutions for Small and Medium
>>> Sized Companies
>>>
>>>
>>
>>
>> --
>> bayoda.com - Professional Online Backup Solutions for Small and Medium
>> Sized Companies
>>
>>
>
>
> --
> bayoda.com - Professional Online Backup Solutions for Small and Medium
> Sized Companies
>
>
>


-- 
bayoda.com - Professional Online Backup Solutions for Small and Medium Sized
Companies

Mime
View raw message