logging-log4j-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remko Popma <remko.po...@gmail.com>
Subject Re: Fastest appender for local structured storage
Date Wed, 03 Jun 2015 14:35:30 GMT
MBs per minute should not be a problem for a modern HDD. 

See http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

Disk seeks are expensive though, so you'll want to buffer your I/O. Big sequential writes
are relatively cheap. 

Log4j's RandomAccessFileAppender (optionally async to ensure bursts don't slow down your app)
should be sufficient. 

For your data structure you could consider fixed length binary records. This is blazingly
fast. 
In a low-latency trading app I was logging ~100MB/sec (slightly overdoing it, ahem). This
wrote fixed length binary records to a memory mapped file. Worked very well. 

Sent from my iPhone

> On 2015/06/03, at 17:21, Gary Gregory <garydgregory@gmail.com> wrote:
> 
> I have a use case where I want an appender to write to local storage (a file), the file
should be structured, such that I can query it later (or load it into a database). Since I
will log a lot of data very fast (possibly MBs a minute), I might or might not use log4j in
async mode. But the bottom line is that I'd like to log to local structured storage without
paying the cost of going through a database layer (NoSQL, JDBC) or a socket. This makes me
wonder if I should create a CSV file appender... which would be easy enough.
> 
> Does anyone here have experience with a use case like this? I'm not crazy about running
MongoDB on the side just to gather logging, but maybe that's what I'll need to do...
> 
> Thoughts?
> 
> Gary
> 
> -- 
> E-Mail: garydgregory@gmail.com | ggregory@apache.org 
> Java Persistence with Hibernate, Second Edition
> JUnit in Action, Second Edition
> Spring Batch in Action
> Blog: http://garygregory.wordpress.com 
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory

Mime
View raw message