karaf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Schneider <ch...@die-schneider.net>
Subject Some ideas for decanter
Date Mon, 09 Mar 2015 09:45:45 GMT
I finally managed to take a close look into decanter this weekend.

I really like the idea of managing logging events as a Map<String, 
Object>. It allows a lot of flexibiltiy while giving the events a clear 
key/value structure that is very near
how key/value stores work. Maybe we need to specify some key names for 
typical data like the timestamp to make the format clearer but in 
general I like that it can transport almost every simple data.

In general the reason why I looked into decanter is that I am searching 
for a logging solution that covers technical logging (like the karaf log 
file) and business logging (events that are directly in the vocabulary 
of the business users). Additionally the solution should not require a 
lot of configuration.

So in slf4j / log4j a nice way to implement business logging is to us 
the MDC. It allows to add arbitrary key/value pairs to logging messages. 
So it is easy to add for example a customer id or a contract id to 
logging messages. This allows to use tools like kibana to look into 
transactions of a specific user.

For the cases where you do not want to use the logging framework OSGi 
events would make a perfect match to transports Map data while providing 
loose coupling.

So when I looked into decanter with the above requirements / ideas I 
found the following place where we could improve it:

1. Structure of the Appender data
I think the structure of the Appender data is a bit awkward:
     Map<Long, Map<String, Object>> data
This means that we can not transport data if it happens at the same Date 
which in practice could happen. So if we need several events we should 
rather use Collection<Map<String, Object>>
I am not sure if we really need several events in one call. The 
Appenders could do this internally so I would rather have the data be 
just a simple Map<String, Object>

2. Use of the EventAdmin service
If we just need to transport a Map then why not simply use the OSGi 
EventAdmin. It transports a Map<String, Object> on a topic. The topic is 
even very similar to logging categories.
EventAdmin is standardized so we do not have to invent a new interface. 
An appender would then simply subscribe to a topic and process the Events.
A collector would simply create and send EventAdmin events.
The EventAdmin already provides a nice implementation of the Dispatcher.

3. Logging of MDC data
The current appender.log does not add the mdc Map to the log data. I 
plan to add this to better cover the business logging requirement. One 
question here is of we should rather flatten the MDC values using a 
prefix on the key or rather put the whole MDC map into the data map 
under one key "mdc". At least elastic search can process such nested maps.

4. How to send to elastic search.
The current appender.elasticsearch uses the elasticsearch API. I propose 
to use JestClient instead. JestClient uses the HttpRest interface of 
elastic search and can also be configured for Authentication. It should 
have a smaller footprint then full elastic search.

5. The decanter feature is version 1.3 of the xsd and seems to fail in 
karaf < 4
So I propose to use the older feature definition to make sure we can 
also cover at least karaf 3.


WDYT?

Btw. I also plan to improve the CXF request / response logging to 
extract meta data from Rest services too and map the meta data to MDC 
key/values. So this would work very well with decanter (with MDC 
improvement). With just very little configuration all meta data could be 
transported to elastic search and anaylized nicely.

Christian

-- 
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com


Mime
View raw message