Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: message received from 54.164.171.186
 which is an MX secondary for users@activemq.apache.org)
MIME-Version: 1.0
Sender: burtonator2011@gmail.com
From: Kevin Burton <burton@spinn3r.com>
Date: Sun, 19 Apr 2015 17:05:54 -0700
Message-ID: 
 <CAAZU44nY0dhfeqrujEqvbh5TAcdzKoyP=dM69AEmvwdycGiAMg@mail.gmail.com>
Subject: Storing message off heap, message compression, and storing the whole
 message as headers.
To: users@activemq.apache.org
Content-Type: multipart/alternative; boundary=001a1138e442c35b9105141cb2ca

--001a1138e442c35b9105141cb2ca
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I=E2=80=99ve been thinking about how messages are stored in the broker and =
ways to
improve the storage in memory.

First, right now, messages are stored in the same heap, and if you=E2=80=99=
re using
the memory store, like, that=E2=80=99s going to add up.  This will increase=
 GC
latency , and you actually need 2x more memory because you have to have
temp memory set aside for GCs.

I was thinking about using Chronicle to store the messages off heap using
direct buffers.  The downside to this is that the messages need to be
serialized/deserialized with each access. But realistically that=E2=80=99s =
probably
acceptable because you can do something like 1M message deserializations
per second.  Which is normally more than the throughput of the broker.

Additionally, chronicle supports zlib or snappy compression on the message
bodies.  So, while the broker supports message compression now, it doesn=E2=
=80=99t
support this feature on headers.

This would give us header compression!

The broker would transparently decompress the headers when reading the
message.

This then begs the question, why use message bodies at all?  Why not just
store an entire message as a set of headers?

If you need hierarchy you can do foo.bar.cat.dog style header names.


--=20

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
=E2=80=A6 or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

--001a1138e442c35b9105141cb2ca--