lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <>
Subject A Fast, Generic, Solr Log Reader
Date Sat, 30 Aug 2014 00:35:17 GMT
I am often asked to take a look at one to many Solr log files that are
hundreds of megabytes to gigabytes in size. "Peaking" at this amount of
logs is a bit time consuming. Anybody that does this often enough has to
build a log parsing tool eventually. One off greps can only get you so far.
The last log reader I started putting together was after a hurricane a year
or two back while I was without power for a week. I have a 6 core machine
and a lot of RAM and I wanted to be able to blow through hundreds of
megabytes of log files in a few seconds (why not). So I started this multi
threaded single file Solr log reader that uses MappedByteBuffers. As I have
had to do a little debugging here or there over the months I have added a
bit to the program. It's my current go to tool when faced with a dozen
gigabyte log files. And it's pretty darn fast if you have some cores to
throw at it.

At this point, to avoid losing the program and someday starting yet again,
I've shared it on GitHub and invite anyone else looking to have a really
good standard log analyzer available to help complete it or offer logs for
testing. It's still early. It's still basically in personal, hack project
phase. But as I have further needs I will continue to add to it and improve
it. It already does a bit more than the current sample output in the README.

My goal is to make it generic enough to read a variety of formats – either
by being flexible internally or through user configuration. With a little
effort, there is a lot of great information and summarization that can be
pulled out of Solr logs.

- Mark

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message