cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terje Marthinussen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-4175) Reduce memory, disk space, and cpu usage with a column name/id map
Date Wed, 03 Jul 2013 07:20:23 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698701#comment-13698701
] 

Terje Marthinussen edited comment on CASSANDRA-4175 at 7/3/13 7:19 AM:
-----------------------------------------------------------------------

Hi, 

Sorry for the late update.

Yes, we have a cluster with some 20-30 billion columns (maybe even closer to 40 billion by
now) which implements a column name map and has been in production for about 2 years.

I was actually looking at committing this 2 years ago together with fairly large number of
other changes which was implemented in the column/supercolumn serializer code but I never
got  around to implement a good way to push the sstable version numbers into the serializer
to make things backwards compatible before focus moved resources elsewhere.

As mentioned above by others, while not benchmarked and proven, I had a very good feeling
the total change helped quite a bit on GC issues, memtables and a bit on performance in general,
but in terms of disk space, the benefit was somewhat limited after sstable compression was
implemented as the repeating column names are compressed pretty well.

This is already 2 years ago (the cluster still runs by the way), but if memory serves me right:
30-40% reduction in disk space without compression
10% reduction on top of compression (I did a test after it was implemented).

In my case, the implementation is actually hardcoded due to time constraints. 
A static map which is global for the entire cassandra installation.

If committing this into cassandra, I believe my plan was split in 3.
Possible as 3 different implementation stages:

1. A simple config option (as a config file or as a columnfamily) where users themselves can
assign repeating column names. Sure, it is not as fancy as many other options, but maybe we
could open up to cover some strange corner case usages here with things like substrings as
well.

Think options to cover complex versions of patterns like date/times such as 20130701202020
where a large chunk of the column name repeats, but not all of it.

In the current implementation, if there is a mapping entry, it converts the string to a variable
length integer which becomes the new column name. If there is no mapping entry, it stores
the raw data.

In our case, we have <40 repeating column names so I never need more than a 1 byte varint.

I also modified the column format to add a "column feature bitmap" at the start of each column.
This allowed me to turn on/off name/id mapping as well as things like TTL's and a handful
of other meta data.

There is a bunch of 64 bit numbers in the column format which only have default value in 99.999%
of all cases and very often your column value is just an 8 byte int, a boolean or a short
text entry. That is, in most cases the column meta data is many times larger than the value
stored.

This would have been my first implementation. Mostly because I have a working implementation
of it already and the mapping table would be very easy to move to a config file with just
a list of column names read at cassandra startup, or stored in a similar way to column family
and other internal config (just as another keyspace for config). Unfortunately, it is a little
bit work also to push such config data down to the serializer. At least as the code was organized
2 years ago.

Notice again, you do not need any sort of atomic handling of the updates to the map in any
way in this implementation. You can add map entries at any time. The result after deserializing
is always the same as column names can have a mix of raw and map id values thanks to the "column
feature bitmap" that was introduced.

Entries that was stored as raw strings will eventually be replaced by ID's to the map as compaction
clean things up.

2. Auto learning feature with mapping table per sstable. 
This would be stage 2 of the implementation.

When starting to create a new SSTable, build a sampling of the most frequently occuring column
names and gradually start mapping them to ID's.

Add the mapping table to the end of the SSTable or in a separate .map file (similar to index
files) at the completion of sstable generation.

The initial id mapping could be further improved by maintaining a global map of column names.
This "global map" would not be used for serialization/deserialization. It would be used to
pre-populate the value for a sstable and would only be statistics to optimize things further
by reducing the number of mapping variances between sstables and reducing the number of raw
values getting stored a bit more.

The id map would still be local to each sstable in terms of storage, but having such statistics
would allow you to dramatically reduce the size of a potentially shared id cache across sstables
where a lot of mapping entries would be identical.

Some may feel that we would run out of memory quickly or use a lot of extra disk with maps
per sstable, but I guess that we only really need to deal with the top few thousand entries
in each sstable and this would not be a problem to keep in a idmap cache in terms of size.

This is really just the top X re-occuring column names or column name sub pattern

If you have more unique column entries that this in a sstable, this will probably not be the
feature that will save the day anyway as the benefit per column entry will be quite small
vs. the overhead and the entire feature should potentially disable itself automagically if
there is no frequently repeating patterns.

3. I had some ideas for moving the mapping up from the serializer to allow things like streaming
entries including id maps between nodes, but things do indeed quickly get ugly and I do not
remember clearly how I had planned to do this.

---
The reason I isolated the mapping function to the serializer is that it looked incredibly
messy to move this further "up" in the stack. Column sorts, range scans, lookukups... 

Not fun at all and if the memtable is serialized anyway the memory consumption there and in
disk cache is dramatically reduced.

Also... with a global static map here at startup time, I actually share the mapped strings
across most columns in memory anyway as I believe they all become pointers to my static complied
in map (again, this gets a lot more trivial to make work very well if this is a startup config,
but yes a bit less user unfriendly)

I haven't looked at the cassandra code for way to long now.

Has it become easier to get to know sstable version numbers in the serializer class now?

I could maybe check if someone in the team here would like to take a stab at moving this to
latest cassandra and commit it if the above implementation seems interesting. 

Part of it should be really easy to port as long as we can get a bit more info into the serializer/deserializer.

                
      was (Author: terjem):
    Hi, 

Sorry for the late update.

Yes, we have a cluster with some 20-30 billion columns (maybe even closer to 40 billion by
now) which implements a column name map and has been in production for about 2 years.

I was actually looking at committing this 2 years ago together with fairly large number of
other changes which was implemented in the column/supercolumn serializer code but I never
got  around to implement a good way to push the sstable version numbers into the serializer
to make things backwards compatible before focus moved resources elsewhere.

As mentioned above by others, while not benchmarked and proven, I had a very good feeling
the total change helped quite a bit on GC issues, memtables and a bit on performance in general,
but in terms of disk space, the benefit was somewhat limited after sstable compression was
implemented as the repeating column names are compressed pretty well.

This is already 2 years ago (the cluster still runs by the way), but if memory serves me right:
30-40% reduction in disk space without compression
10% reduction on top of compression (I did a test after it was implemented).

In my case, the implementation is actually hardcoded due to time constraints. A static map
which is global for the entire cassandra installation.

If committing this into cassandra, I believe my plan was split in 3.
Possible as 3 different implementation stages:

1. A simple config option (as a config file or as a columnfamily) where users themselves can
assign repeating column names. Sure, it is not as fancy as many other options, but maybe we
could open up to cover some strange corner case usages here with things like substrings as
well.

Think options to cover complex versions of patterns like date/times such as 20130701202020
where a large chunk of the column name repeats, but not all of it.

In the current implementation, if there is a mapping entry, it converts the string to a variable
length integer which becomes the new column name. If there is no mapping entry, it stores
the raw data.

In our case, we have <40 repeating column names so I never need more than 1 byte, but the
implementation would handle more if I had.

I modified the sstable to add a bitmap at the start of each column to be able to turn on/off
mapping entries, timestamps not used, TTL's and other things. There is a bunch of 64 bit numbers
in the column format which only have default value in 99.999% of all cases and very often
your column value is just an 8 byte int, a boolean or a short text entry. 

I think in 99% of the columns in this cassandra store, the column timestamp takes up more
space than the column value.

This would have been my first implementation. Mostly because I have a working implementation
of it already and the mapping table would be very easy to move to a config file read at start
of a column family similar to what we have for CF config but also here, it is a bit work to
push such config data down to the serializer as the code was organized 2 years ago.

Notice again, you do not need atomic handling of the updates to the map in any way in this
implementation. You can add map entries at any time. The result after deserializing is always
the same as column names can have a mix of raw and map id values thanks to the "column feature
bitmap" that was introduced.

2. Auto learning feature with mapping table per sstable. 
This would be stage 2 of the implementation.

When starting to create a new SSTable, build a sampling of the most frequently occuring column
names and gradually start mapping them to ID's.

Add the mapping table to the end of the SSTable or in a separate .map file (similar to index
files) at the completion of sstable generation.

The initial id mapping could be further improved by maintaining a global map of column names.
This "global map" would not be used for serialization/deserialization. It would be used to
pre-populate the value for a sstable and would only be statistics to optimize things further
by reducing the number of mapping variances between sstables and reducing the number of raw
values getting stored a bit more.

The id map would still be local to each sstable in terms of storage, but having such statistics
would allow you to dramatically reduce the size of a potentially shared id cache across sstables
where a lot of mapping entries would be identical.

Some may feel that we would run out of memory quickly or use a lot of extra disk with maps
per sstable, but I guess that we only really need to deal with the top few thousand entries
in each sstable and this would not be a problem to keep in a idmap cache in terms of size.

This is really just the top X re-occuring column names or column name sub pattern

If you have more unique column entries that this in a sstable, this will probably not be the
feature that will save the day anyway as the benefit per column entry will be quite small
vs. the overhead and the entire feature should potentially disable itself automagically if
there is no frequently repeating patterns.

3. I had some ideas for moving the mapping up from the serializer to allow things like streaming
entries including id maps between nodes, but things do indeed quickly get ugly and I do not
remember clearly how I had planned to do this.

---
The reason I isolated the mapping function to the serializer is that it looked incredibly
messy to move this further "up" in the stack. Column sorts, range scans, lookukups... 

Not fun at all and if the memtable is serialized anyway the memory consumption there and in
disk cache is dramatically reduced.

Also... with a global static map here at startup time, I actually share the mapped strings
across most columns in memory anyway as I believe they all become pointers to my static complied
in map (again, this gets a lot more trivial to make work very well if this is a startup config,
but yes a bit less user unfriendly)

I haven't looked at the cassandra code for way to long now.

Has it become easier to get to know sstable version numbers in the serializer class now?

I could maybe check if someone in the team here would like to take a stab at moving this to
latest cassandra and commit it if the above implementation seems interesting. 

Part of it should be really easy to port as long as we can get a bit more info into the serializer/deserializer.

                  
> Reduce memory, disk space, and cpu usage with a column name/id map
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-4175
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4175
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>             Fix For: 2.1
>
>
> We spend a lot of memory on column names, both transiently (during reads) and more permanently
(in the row cache).  Compression mitigates this on disk but not on the heap.
> The overhead is significant for typical small column values, e.g., ints.
> Even though we intern once we get to the memtable, this affects writes too via very high
allocation rates in the young generation, hence more GC activity.
> Now that CQL3 provides us some guarantees that column names must be defined before they
are inserted, we could create a map of (say) 32-bit int column id, to names, and use that
internally right up until we return a resultset to the client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message