activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nigro_franz <nigro....@gmail.com>
Subject Re: [DISCUSS] Memory Mapped Journal Type
Date Fri, 03 Feb 2017 08:58:10 GMT
@clebertsuconic But even if is a personal request, IMHO would be really great
to have a separate (open source of course!) project that aims to provide a
journal as a first class citizen, maybe with pluggable features like
different binary encoders/decoders (SBE, ProtoBuf, FlatBuffers..),
replication/compaction/paging and using different stores (redis, SQL,
cassandra, elasticsearch, Kafka)...the possibilities could be infinite,
wdyt?

> when  you do sync=false, is that actually writing to the 
> disk, or it's just putting it in memory until you call force or close 
> the disk? 

AFAIK it is really writing to the disk, but asynchronously and using the OS
configuration for it (on linux: numa balancing, swappiness,
dirty_expire_centisecs, dirty_ratio, dirty_writeback_centisecs and many
more).

> I'm not arguing how safe is to use either, but on my crash tests the 
> messages weren't lost if I waited some time before killing the server. 

Cool! Effectively It's guaranteed to be safe with process failure, indeed
AFAIK pretty almost any OS that provides mmap ensures that every writes
performed on mapped memory will be persisted on physical storage even if the
owner process will crash! The only way to force a mmap file to lose data is
with power failures/OS hard-crashes (that sadly Murphy's law said that will
happen for sure eheh).

> IAIO with no-sync probably has a higher level of guarantees of being 
>> actually on the disk in case of crashes.. but if the user don't need 
>> syncs, that's a rock & roll possibility.

Yes, i think the same, it has a good balance between reliability (on process
crashes, but on the cloud it could be enough :P) and speed but AIO is
unbeatable from this POV: if the write is reported as performed it is no way
on the disk, even on OS failures!
Another 2 points of difference will be memory and cpu usage: the writes
you're seeing are flushed by the OS on the disk in background, using the
virtual memory directly as a buffer (but on lAIO there is no need to use
it!); indeed the CPU usage will be, for high writes rate, a little higher
than the lAIO version, due to this async flushes, right?

> Take a look on my branch, mapped at my fork

Thanks!For sure!!!



--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Memory-Mapped-Journal-Type-tp4721494p4721560.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Mime
View raw message