activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nigro_franz <nigro....@gmail.com>
Subject Re: Adapting TimedBuffer and NIO Buffer Pooling
Date Tue, 09 May 2017 09:32:20 GMT
> I quite liked the quiver tool that was shared to me a while back I think
Clebert shared a link, very similar to our internal testing rig, so much so
I started to use it instead of our own especially in non Java client testing
:) 

+100 quiver is a very good polyglot benchmarking tool and is pretty simple
to use too, hence to built an hypothetical end user testing chain is by far
the best candidate I know.
But the reasons why I've provided an additional custom tool is:
- to benchmark responsiveness under load and all out throughput benchmarks
(see
http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html)
and account for coordinated omission issues
- to collect all the possible latencies in real time or/and optionally
histograms (HdrHstogram)
- to have proper compiled methods (server-side and client-side) in a way
similar to JMH (JITWatch is a great tool to find these infos)

So the tool I've made is not for an end user tool but a (broker side)
developer one: different the scopes, different the means.

>Re SSD's my personal feeling is that this is the way of the data centre
(hdd will start being relegated to glacial or non transactional storage) 
Agree, but to exploit the parallelism of these devices is required a proper
application domain too: key-value stores can be partitioned naturally in
different ways (see ChronicleMap/Hazelcast etc etc) while WAL "depends".
Hence to implement fast/better WAL right now the safer (and faster ie LMDB)
choice seems to design it in order to exploit sequential patterns, exactly
as is possible to be done with a normal HDDs. 
I postpone other impressions on it when I'll be more skilled on how SSD
works and to make better use of them, but the first limit to scale seems
more a domain partitioning issue than a physical one.

> For me it's about parallelism and stop treating SSD like a HDD
Agree, but must be considered this too:
http://www.perfdynamics.com/Manifesto/USLscalability.html
Being parallel is possible, but has a cost without reducing concurrency
(conceptually first) IMHO.

Thanks for the references, are cool! I'm looking to LMDB, Neo4J and
Cassandra too, but AFAIK even if they ensure durability , currently the best
performing choices around transactions seems to have fixed background
checkpoints (maybe with IO limiters, like Neo4J).

> Also keep in mind the new features up coming like non volatile ram with
> optane 3D xpoint 
About pmem and 3D Xpoints, I've tried to move on it some time ago:
https://groups.google.com/forum/#!topic/pmem/2LJFFpoc8gA
But seems early to have compatible implementations in Java that could be
used like the memory mapped files (with compatible API I mean): anyway the
original reason behind the MAPPED journal was to be ready/familiar with them
and their candidate APIs.











--
View this message in context: http://activemq.2283324.n4.nabble.com/Adapting-TimedBuffer-and-NIO-Buffer-Pooling-tp4725727p4725819.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Mime
View raw message