logging-log4j-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Gregory <garydgreg...@gmail.com>
Subject Re: In memory appender
Date Tue, 27 Sep 2016 01:09:38 GMT
On Mon, Sep 26, 2016 at 5:21 PM, Ralph Goers <ralph.goers@dslextreme.com>
wrote:

> I thought you didn’t want to write to a file?
>

I do not but if the buffer is large enough, log events should stay in RAM.
But it is not quite right anyway because I'd have to interpret the contents
of the file to turn back into log events.

I started reading up on the Chronicle appender; thank you Remko for
pointing it out.

An appender to a cache of objects is really want I want since I also want
to be able to evict the cache. TBC...

Gary


> The Chronicle stuff Remko is linking to is also worth exploring.
>
> Ralph
>
>
>
> On Sep 26, 2016, at 5:04 PM, Gary Gregory <garydgregory@gmail.com> wrote:
>
> oh... what about our own http://logging.apache.org/
> log4j/2.x/manual/appenders.html#MemoryMappedFileAppender
>
> ?
>
> Gary
>
> On Mon, Sep 26, 2016 at 4:59 PM, Remko Popma <remko.popma@gmail.com>
> wrote:
>
>> In addition to the Flume based solution, here is another alternative
>> idea: use Peter Lawrey's Chronicle[1] library to store log events in a
>> memory mapped file.
>>
>> The appender can just keep adding events without worrying about
>> overflowing the memory.
>>
>> The client that reads from this file can be in a separate thread (even a
>> separate process by the way) and can read as much as it wants, and send it
>> to the server.
>>
>> Serialization: You can either serialize log events to the target format
>> before storing them in Chronicle (so you have binary blobs in each
>> Chronicle excerpt), client reads these blobs and sends them to the server
>> as is. Or you can use the Chronicle Log4j2 appender[2] to store the events
>> in Chronicle format. The tests[3] show how to read LogEvent objects from
>> the memory mapped file, and the client would be responsible for serializing
>> these log events to the target format before sending data to the server.
>>
>> [1]: https://github.com/peter-lawrey/Java-Chronicle
>> [2]: https://github.com/OpenHFT/Chronicle-Logger
>> [3]: https://github.com/OpenHFT/Chronicle-Logger/blob/master
>> /logger-log4j-2/src/test/java/net/openhft/chronicle/logger/
>> log4j2/Log4j2IndexedChronicleTest.java
>>
>> Remko
>>
>> Sent from my iPhone
>>
>> On 2016/09/27, at 5:57, Gary Gregory <garydgregory@gmail.com> wrote:
>>
>> Please allow me to restate the use case I have for the
>> CollectionAppender, which is separate from any Flume-based or Syslog-based
>> solution, use cases I also have. Well, I have a Syslog use case, and
>> whether or not Flume is in the picture will really be a larger discussion
>> in my organization due to the requirement to run a Flume Agent.)
>>
>> A program (like a JDBC driver already using Log4j) communicates with
>> another (like a DBMS, not written in Java). The client and server
>> communicate over a proprietary socket protocol. The client sends a list of
>> buffers (in one go) to the server to perform one or more operations. One
>> kind of buffer this protocol defines is a log buffer (where each log event
>> is serialized in a non-Java format.) This allows each communication from
>> the client to the server to say "This is what's happened up to now". What
>> the server does with the log buffers is not important for this discussion.
>>
>> What is important to note is that the log buffer and other buffers go to
>> the server in one BLOB; which is why I cannot (in this use case) send log
>> events by themselves anywhere.
>>
>> I see that something (a CollectionAppender) must collect log events until
>> the client is ready to serialize them and send them to the server. Once the
>> events are drained out of the Appender (in one go by just getting the
>> collection), events can collect in a new collection. A synchronous drain
>> operation would create a new collection and return the old one.
>>
>> The question becomes: What kind of temporary location can the client use
>> to buffer log event until drain time? A Log4j Appender is a natural place
>> to collect log events since the driver uses Log4j. The driver will make its
>> business to drain the appender and work with the events at the right time.
>> I am thinking that the Log4j Appender part is generic enough for inclusion
>> in Log4j.
>>
>> Further thoughts?
>>
>> Thank you all for reading this far!
>> Gary
>>
>> On Sun, Sep 25, 2016 at 1:20 PM, Ralph Goers <ralph.goers@dslextreme.com>
>> wrote:
>>
>>> I guess I am not understanding your use case quite correctly. I am
>>> thinking you have a driver that is logging and you want those logs
>>> delivered to some other location to actually be written.  If that is your
>>> use case then the driver needs a log4j2.xml that configures the
>>> FlumeAppender with either the memory or file channel (depending on your
>>> needs) and points to the server(s) that is/are to receive the events. The
>>> FlumeAppender handles sending them in batches with whatever size you want
>>> (but will send them in smaller amounts if they are in the channel too
>>> long). Of course you would need the log4j-flume and flume jars. So on the
>>> driver side you wouldn’t need to write anything, just configure the
>>> appender and make sure the jars are there.
>>>
>>> For the server that receives them you would also need Flume. Normally
>>> this would be a standalone component, but it really wouldn’t be hard to
>>> incorporate it into some other application. The only thing you would have
>>> to write would be the sink that writes the events to the database or
>>> whatever. To incorporate it into an application you would have to look at
>>> the main() method of flume and covert that to be a thread that you kick off.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Sep 25, 2016, at 12:01 PM, Gary Gregory <garydgregory@gmail.com>
>>> wrote:
>>>
>>> Hi Ralph,
>>>
>>> Thanks for your feedback. Flume is great in the scenarios that do not
>>> involve sending a log buffer from the driver itself.
>>>
>>> I can't require a Flume Agent to be running 'on the side' for the use
>>> case where the driver chains a log buffer at the end of the train of
>>> database IO buffer. For completeness talking about this Flume scenario, if
>>> I read you right, I also would need to write a custom Flume sink, which
>>> would also be in memory, until the driver is ready to drain it. Or, I could
>>> query some other 'safe' and 'reliable' Flume sink that the driver could
>>> then drain of events when it needs to.
>>>
>>> Narrowing down on the use case where the driver chains a log buffer at
>>> the end of the train of database IO buffer, I'll think I have to see about
>>> converting the Log4j ListAppender into a more robust and flexible version.
>>> I think I'll call it a CollectionAppender and allow various Collection
>>> implementations to be plugged in.
>>>
>>> Gary
>>>
>>> Gary
>>>
>>> On Sat, Sep 24, 2016 at 3:44 PM, Ralph Goers <ralph.goers@dslextreme.com
>>> > wrote:
>>>
>>>> If you are buffering events in memory you run the risk of losing events
>>>> if something should fail.
>>>>
>>>> That said, if I had your requirements I would use the FlumeAppender. It
>>>> has either an in-memory option to buffer as you are suggesting or it can
>>>> write to a local file to prevent data loss if that is a requirement. It
>>>> already has the configuration options you are looking for and has been well
>>>> tested. The only downside is that you need to have either a Flume instance
>>>> receiving the messages are something that can receive Flume events over
>>>> Avro, but it is easier just to use Flume and write a custom sink to do what
>>>> you want with the data.
>>>>
>>>> Ralph
>>>>
>>>> On Sep 24, 2016, at 3:13 PM, Gary Gregory <garydgregory@gmail.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I can't believe it, but through a convoluted use-case, I actually need
>>>> an in-memory list appender, very much like our test-only ListAppender.
>>>>
>>>> The requirement is as follows.
>>>>
>>>> We have a JDBC driver and matching proprietary database that
>>>> specializes in data virtualization of mainframe resources like DB2, VSAM,
>>>> IMS, and all sorts of non-SQL data sources (
>>>> http://www.rocketsoftware.com/products/rocket-data/rocket-d
>>>> ata-virtualization)
>>>>
>>>> The high level requirement is to merge the driver log into the server's
>>>> log for full-end to end tractability and debugging.
>>>>
>>>> When the driver is running on the z/OS mainframe, it can be configured
>>>> with a z/OS specific Appender that can talk to the server log module
>>>> directly.
>>>>
>>>> When the driver is running elsewhere, it can talk to the database via a
>>>> Syslog socket Appender. This requires more set up on the server side and
>>>> for the server to do special magic to know how the incoming log events
>>>> match up with server operations. Tricky.
>>>>
>>>> The customer should also be able to configure the driver such that
>>>> anytime the driver communicates to the database, it sends along whatever
>>>> log events have accumulated since the last client-server roundtrip. This
>>>> allows the server to match exactly the connection and operations the client
>>>> performed with the server's own logging.
>>>>
>>>> In order to do that I need to buffer all log events in an Appender and
>>>> when it's time, I need to get the list of events and reset the appender to
>>>> a new empty list so events can keep accumulating.
>>>>
>>>> My proposal is to either turn our ListAppender into such an appender.
>>>> For sanity, the appender could be configured with various sizing policies:
>>>>
>>>> - open: the list grows unbounded
>>>> - closed: the list grows to a given size and _new_ events are dropped
>>>> on the floor beyond that
>>>> - latest: the list grows to a given size and _old_ events are dropped
>>>> on the floor beyond that
>>>>
>>>> Thoughts?
>>>>
>>>> Gary
>>>>
>>>> --
>>>> E-Mail: garydgregory@gmail.com | ggregory@apache.org
>>>> Java Persistence with Hibernate, Second Edition
>>>> <http://www.manning.com/bauer3/>
>>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>>>> Spring Batch in Action <http://www.manning.com/templier/>
>>>> Blog: http://garygregory.wordpress.com
>>>> Home: http://garygregory.com/
>>>> Tweet! http://twitter.com/GaryGregory
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> E-Mail: garydgregory@gmail.com | ggregory@apache.org
>>> Java Persistence with Hibernate, Second Edition
>>> <http://www.manning.com/bauer3/>
>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>>> Spring Batch in Action <http://www.manning.com/templier/>
>>> Blog: http://garygregory.wordpress.com
>>> Home: http://garygregory.com/
>>> Tweet! http://twitter.com/GaryGregory
>>>
>>>
>>>
>>
>>
>> --
>> E-Mail: garydgregory@gmail.com | ggregory@apache.org
>> Java Persistence with Hibernate, Second Edition
>> <http://www.manning.com/bauer3/>
>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>> Spring Batch in Action <http://www.manning.com/templier/>
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> Java Persistence with Hibernate, Second Edition
> <http://www.manning.com/bauer3/>
> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> Spring Batch in Action <http://www.manning.com/templier/>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>
>
>


-- 
E-Mail: garydgregory@gmail.com | ggregory@apache.org
Java Persistence with Hibernate, Second Edition
<http://www.manning.com/bauer3/>
JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
Spring Batch in Action <http://www.manning.com/templier/>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Mime
View raw message