Mailing-List: contact log4j-dev-help@logging.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Log4J Developers List" <log4j-dev@logging.apache.org>
Subject: Re: Fastest appender for local structured storage
References: 
 <CACZkXPzAFZvjo4nXYom6Uk1WC+Vb=Y5x2Oco+PFDBBqY6KcAdA@mail.gmail.com>
From: Remko Popma <remko.popma@gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-2B488209-3A6D-4FF0-A523-4790C689C7A5
In-Reply-To: 
 <CACZkXPzAFZvjo4nXYom6Uk1WC+Vb=Y5x2Oco+PFDBBqY6KcAdA@mail.gmail.com>
Message-Id: <7D929D35-9E49-4833-8681-153887EFAEE0@gmail.com>
Date: Wed, 3 Jun 2015 23:35:30 +0900
To: Log4J Developers List <log4j-dev@logging.apache.org>
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)

--Apple-Mail-2B488209-3A6D-4FF0-A523-4790C689C7A5
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable

MBs per minute should not be a problem for a modern HDD.=20

See http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

Disk seeks are expensive though, so you'll want to buffer your I/O. Big sequ=
ential writes are relatively cheap.=20

Log4j's RandomAccessFileAppender (optionally async to ensure bursts don't sl=
ow down your app) should be sufficient.=20

For your data structure you could consider fixed length binary records. This=
 is blazingly fast.=20
In a low-latency trading app I was logging ~100MB/sec (slightly overdoing it=
, ahem). This wrote fixed length binary records to a memory mapped file. Wor=
ked very well.=20

Sent from my iPhone

> On 2015/06/03, at 17:21, Gary Gregory <garydgregory@gmail.com> wrote:
>=20
> I have a use case where I want an appender to write to local storage (a fi=
le), the file should be structured, such that I can query it later (or load i=
t into a database). Since I will log a lot of data very fast (possibly MBs a=
 minute), I might or might not use log4j in async mode. But the bottom line i=
s that I'd like to log to local structured storage without paying the cost o=
f going through a database layer (NoSQL, JDBC) or a socket. This makes me wo=
nder if I should create a CSV file appender... which would be easy enough.
>=20
> Does anyone here have experience with a use case like this? I'm not crazy a=
bout running MongoDB on the side just to gather logging, but maybe that's wh=
at I'll need to do...
>=20
> Thoughts?
>=20
> Gary
>=20
> --=20
> E-Mail: garydgregory@gmail.com | ggregory@apache.org=20
> Java Persistence with Hibernate, Second Edition
> JUnit in Action, Second Edition
> Spring Batch in Action
> Blog: http://garygregory.wordpress.com=20
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory

--Apple-Mail-2B488209-3A6D-4FF0-A523-4790C689C7A5
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div>MBs per minute should not be a problem=
 for a modern HDD.&nbsp;</div><div><br></div><div>See&nbsp;<a href=3D"http:/=
/www.eecs.berkeley.edu/~rcs/research/interactive_latency.html">http://www.ee=
cs.berkeley.edu/~rcs/research/interactive_latency.html</a></div><div><br></d=
iv><div>Disk seeks are expensive though, so you'll want to buffer your I/O. B=
ig sequential writes are relatively cheap.&nbsp;</div><div><br></div><div>Lo=
g4j's RandomAccessFileAppender (optionally async to ensure bursts don't slow=
 down your app) should be sufficient.&nbsp;</div><div><br></div><div>For you=
r data structure you could consider fixed length binary records. This is bla=
zingly fast.&nbsp;</div><div>In a low-latency trading app I was logging ~100=
MB/sec (slightly overdoing it, ahem). This wrote fixed length binary records=
 to a memory mapped file. Worked very well.&nbsp;</div><div><br>Sent from my=
 iPhone</div><div><br>On 2015/06/03, at 17:21, Gary Gregory &lt;<a href=3D"m=
ailto:garydgregory@gmail.com">garydgregory@gmail.com</a>&gt; wrote:<br><br><=
/div><blockquote type=3D"cite"><div><div dir=3D"ltr">I have a use case where=
 I want an appender to write to local storage (a file), the file should be s=
tructured, such that I can query it later (or load it into a database). Sinc=
e I will log a lot of data very fast (possibly MBs a minute), I might or mig=
ht not use log4j in async mode. But the bottom line is that I'd like to log t=
o local structured storage without paying the cost of going through a databa=
se layer (NoSQL, JDBC) or a socket. This makes me wonder if I should create a=
 CSV file appender... which would be easy enough.<div><br></div><div>Does an=
yone here have experience with a use case like this? I'm not crazy about run=
ning MongoDB on the side just to gather logging, but maybe that's what I'll n=
eed to do...</div><div><br></div><div>Thoughts?</div><div><br></div><div>Gar=
y<br clear=3D"all"><div><br></div>-- <br><div class=3D"gmail_signature"><div=
 dir=3D"ltr">E-Mail: <a href=3D"mailto:garydgregory@gmail.com" target=3D"_bl=
ank">garydgregory@gmail.com</a> | <a href=3D"mailto:ggregory@apache.org" tar=
get=3D"_blank">ggregory@apache.org </a><br><a href=3D"http://www.manning.com=
/bauer3/" target=3D"_blank">Java Persistence with Hibernate, Second Edition<=
/a><br><a href=3D"http://www.manning.com/tahchiev/" target=3D"_blank">JUnit i=
n Action, Second Edition</a><br><a href=3D"http://www.manning.com/templier/"=
 target=3D"_blank">Spring Batch in Action</a><br>Blog: <a href=3D"http://gar=
ygregory.wordpress.com/" target=3D"_blank">http://garygregory.wordpress.com<=
/a> <br>Home: <a href=3D"http://garygregory.com/" target=3D"_blank">http://g=
arygregory.com/</a><br>Tweet! <a href=3D"http://twitter.com/GaryGregory" tar=
get=3D"_blank">http://twitter.com/GaryGregory</a></div></div>
</div></div>
</div></blockquote></body></html>=

--Apple-Mail-2B488209-3A6D-4FF0-A523-4790C689C7A5--