activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael André Pearce <michael.andre.pea...@me.com>
Subject Re: Adapting TimedBuffer and NIO Buffer Pooling
Date Mon, 08 May 2017 22:00:12 GMT
In regards to testing I think having end2end tests is more important than area targeted ones.
Obviously these help but i don't think should be the deciding factor.

I quite liked the quiver tool that was shared to me a while back I think Clebert shared a
link, very similar to our internal testing rig, so much so I started to use it instead of
our own especially in non Java client testing :)

I wonder if there's a way to automate it, so that a bunch of perf stats (latency histo and
throughout) can be produced for a build , given the same kit (build agent) After all this
is what an end user cares about.

Obv I wouldn't run it every build but look to have it reserved for pre tag builds or weekly/monthly
to ensure perf isn't regressed and any complicated or perf improvements like yours. Or can
be manually run but easily locally. This way we don't end up with a bunch of one off or bespoke
tests also.


Re SSD's my personal feeling is that this is the way of the data centre (hdd will start being
relegated to glacial or non transactional storage) . Many many apps already are built or tuned
heavily for SSD already. As long as any feature is behind a toggle so it can eve turned off
or on then I see no reason why not to implement features targeted at specific hardware like
ssd's.

My two cents and thoughts.

Re SSD, have a poke around aerospike documents, they got early access to intel optane to tune
their engine (a few google searches can return some interesting details) really I think they're
one of the leaders in this area.
The other one is the well known rocksdb. 

For me it's about parallelism and stop treating SSD like a HDD, enterprise ones come with
capacitor backed ram caches exploiting these, raid controllers and sata add too much overhead
etc. Also keep in mind the new features up coming like non volatile ram with optane 3D xpoint


Heres a less low level but interesting read also.
https://www.oreilly.com/ideas/how-flash-changes-the-design-of-database-storage-engines








Sent from my iPhone

> On 8 May 2017, at 19:12, nigro_franz <nigro.fra@gmail.com> wrote:
> 
> In the meantime, I'm searching good articles to add other disk I/O limiter
> strategies, maybe based on IOPS credits consumption.
> 
> A possible idea about an IOPS limiter: 
> - a single writer/appender/sync thread to perform the real writes & flush
> - a bounded queue of bytes (I've already designed a couple pretty fast) to
> submit any requests 
> - a proper batch strategy on the appender/sync thread based on the current
> IOPS and unflushed writes
> 
> The bounded queue/ringbuffer will be useful to propagate backpressure
> "naturally" to the request producer, bounding the max latency and with
> bounded memory footprint.
> 
> What do you think?
> 
> Right now I've found only this article:
> https://engineering.linkedin.com/blog/2016/05/designing-ssd-friendly-applications-for-better-application-perfo
> But it doesn't take in account sync operations and is specific only for
> SSDs!!
> If any of you have something good on the argument, please share :)
> 
> 
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Adapting-TimedBuffer-and-NIO-Buffer-Pooling-tp4725727p4725780.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message