activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Storing message off heap, message compression, and storing the whole message as headers.
Date Mon, 20 Apr 2015 00:05:54 GMT
I’ve been thinking about how messages are stored in the broker and ways to
improve the storage in memory.

First, right now, messages are stored in the same heap, and if you’re using
the memory store, like, that’s going to add up.  This will increase GC
latency , and you actually need 2x more memory because you have to have
temp memory set aside for GCs.

I was thinking about using Chronicle to store the messages off heap using
direct buffers.  The downside to this is that the messages need to be
serialized/deserialized with each access. But realistically that’s probably
acceptable because you can do something like 1M message deserializations
per second.  Which is normally more than the throughput of the broker.

Additionally, chronicle supports zlib or snappy compression on the message
bodies.  So, while the broker supports message compression now, it doesn’t
support this feature on headers.

This would give us header compression!

The broker would transparently decompress the headers when reading the
message.

This then begs the question, why use message bodies at all?  Why not just
store an entire message as a set of headers?

If you need hierarchy you can do foo.bar.cat.dog style header names.


-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message