Mailing-List: contact dev-help@karaf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@karaf.apache.org
Received-SPF: pass (athena.apache.org: domain of cschneider111@gmail.com
 designates 74.125.82.174 as permitted sender)
Sender: Christian Schneider <cschneider111@gmail.com>
Message-ID: <54FD80DD.1040204@die-schneider.net>
Date: Mon, 09 Mar 2015 12:15:41 +0100
From: Christian Schneider <chris@die-schneider.net>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: dev@karaf.apache.org
Subject: Re: Some ideas for decanter
References: <54FD6BC9.9090407@die-schneider.net>
 <CAD0r13dtR=C-idR-GLZNQX1wgLKitF0KeX5dw1077mt5DTB1Wg@mail.gmail.com>
In-Reply-To: 
 <CAD0r13dtR=C-idR-GLZNQX1wgLKitF0KeX5dw1077mt5DTB1Wg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Some answers inline:

On 09.03.2015 11:32, Achim Nierbeck wrote:
> Hi Christian,
>
> please find my comments inline:
>
>
> 2015-03-09 10:45 GMT+01:00 Christian Schneider <chris@die-schneider.net>:
> 1. Structure of the Appender data
> I think the structure of the Appender data is a bit awkward:
>      Map<Long, Map<String, Object>> data
> This means that we can not transport data if it happens at the same Date
> which in practice could happen. So if we need several events we should
> rather use Collection<Map<String, Object>>
> I am not sure if we really need several events in one call. The Appenders
> could do this internally so I would rather have the data be just a simple
> Map<String, Object>
>
> which appender did you look at?
> The JMX appender is approprate and the timestamp is collected by the
> elasticsearch trigger/appender transformed to the required timestamp field
> (take a look at the changes I did)
The JMX Appender uses System.currentTimeMillis() as key for the outer 
map. So a second event may be created with the same time if the system 
is fast. I think this is broken and the problem is in the Appender API 
not your implementation.
>
>
>> 2. Use of the EventAdmin service
>> If we just need to transport a Map then why not simply use the OSGi
>> EventAdmin. It transports a Map<String, Object> on a topic. The topic is
>> even very similar to logging categories.
>> EventAdmin is standardized so we do not have to invent a new interface. An
>> appender would then simply subscribe to a topic and process the Events.
>> A collector would simply create and send EventAdmin events.
>> The EventAdmin already provides a nice implementation of the Dispatcher.
>>
>>
> I'd rather stick to the JMX appender as the current default one and would
> make sure the others do log accordingly.
There is no JMX appender. What are you talking about?

My proposal is to connect the Collectors and the Appenders using 
EventAdmin instead of a custom interface that looks almost like 
EventAdmin. Except that EventAdmin has a topic name which also would 
make sense for us.
It has nothing to do with JMX.
>
>
>> 3. Logging of MDC data
>> The current appender.log does not add the mdc Map to the log data. I plan
>> to add this to better cover the business logging requirement. One question
>> here is of we should rather flatten the MDC values using a prefix on the
>> key or rather put the whole MDC map into the data map under one key "mdc".
>> At least elastic search can process such nested maps.
>>
>>
> make sure the stuff looks similar to the changes I did with the JMX
> appender.
So you mean the change with adding the attributes as a map into one 
value. Yes that is how I also thought to do it for the MDC.
>> 4. How to send to elastic search.
>> The current appender.elasticsearch uses the elasticsearch API. I propose
>> to use JestClient instead. JestClient uses the HttpRest interface of
>> elastic search and can also be configured for Authentication. It should
>> have a smaller footprint then full elastic search.
>>
>>
> it's the one used for the internal one and does work appropriate. If the
> elastic search is external a logging appender with the standard ELK stack
> (elasticsearch, logstash, kibana) should be sufficient.
I am talking about how to send data to elastic search. Of course I could 
use logstash but why would I need decanter then?
I think we should provide an appender that can be used in production. If 
the elastic search one can not do any authentication then I would not 
recommend to use it.
Jest at least announces that it can be secured and can talk to 
elasticsearch. I did a small prototype and it looks good.
If we can not agree on jest then maybe we could at least have an 
additional Jest appender. So people can choose.
>> 5. The decanter feature is version 1.3 of the xsd and seems to fail in
>> karaf < 4
>> So I propose to use the older feature definition to make sure we can also
>> cover at least karaf 3.
>>
>>
> worked like a charm for me on Karaf 4.
I would like it to work in karaf 3 and 4. Of course it works on karaf 4.


Christian

-- 
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com