apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: Programmatic log4j appender in Apex
Date Mon, 10 Apr 2017 23:43:57 GMT
It will be good to understand a use case where an operator needs a 
specific appender.

IMO, an operator designer defines *what* should be logged and dev-ops 
team defines *where* to log.

Thank you,

Vlad
On 4/10/17 16:27, Munagala Ramanath wrote:
> Yes, I understand, I was just wondering if individual operators could
> define the appenders
> they potentially need at compile time and then the operator callbacks could
> simply
> check the desired runtime condition and add the appropriate appender.
>
> Or are we saying there are scenarios where we absolutely cannot create the
> appender beforehand ?
>
> So broadly speaking, my question is whether the combination of providing
> predefined appenders
> and the PropertyConfigurator capabilities meets the need.
>
> Ram
>
> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <sergey@datatorrent.com>
> wrote:
>
>> Ram,
>>
>> Really the new appender class must extend the abstract class
>> AppenderSkeleton. And in order to add a new appender programmatically in
>> Java, some code in Apex should call the following log4j method:
>>
>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender newAppender)
>>
>> The general idea of my proposal is "*based on some runtime parameter(s) to
>> provide ability to create an appender instance via reflection and add it to
>> the list of active log4j appenders*".
>>
>> Thanks,
>> Sergey
>>
>>
>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <v.rozov@datatorrent.com>
>> wrote:
>>
>>> It will require application recompilation and repackaging. The proposed
>>> functionality is for dev-ops to be able to route application logging to a
>>> preferred destination without recompiling applications. It is run-time
>>> configuration vs compile time hardcoded appender.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 4/10/17 11:23, Munagala Ramanath wrote:
>>>
>>>> You can do it in a trivial derived class without changing the base
>> class.
>>>> Ram
>>>>
>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <v.rozov@datatorrent.com>
>>>> wrote:
>>>>
>>>> Does not the proposal to use Logger.addAppender() requires modifications
>>>>> to used operators code?
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>> On 4/10/17 10:58, Munagala Ramanath wrote:
>>>>>
>>>>> People can currently do this by simply implementing the Appender
>>>>>> interface
>>>>>> and adding it
>>>>>> with Logger.addAppender() in the setup method. Why do we need
>> something
>>>>>> more elaborate ?
>>>>>>
>>>>>> Ram
>>>>>>
>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
>>>>>> sergey@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> The configuration of a log4j appender via log4j configuration file
is
>> a
>>>>>>> static configuration that cannot be disabled/enabled and managed
>>>>>>> dynamically by an application designer. The programmatic approach
>> will
>>>>>>> allow  an application designer to specify which of the available
>> log4j
>>>>>>> appenders should be used for the specific application.
>>>>>>>
>>>>>>> It is not necessary Apex should use the predefined log4j appenders
>>>>>>> only.
>>>>>>> The log4j events contain useful but the very limited number of
>>>>>>> properties
>>>>>>> which values can be printed into output log4j sources. But based
on
>> the
>>>>>>> knowledge of the software product workflow, the custom defined
log4j
>>>>>>> appender can extend a list of predefined output log events properties
>>>>>>> and,
>>>>>>> for instance for Apex, return: node, user name, application name,
>>>>>>> application id, container id, operator name, etc.
>>>>>>>
>>>>>>> Also the output log events that are generated by a custom defined
>> log4j
>>>>>>> appender can be stored and indexed by any type of a full text
search
>>>>>>> database. It will allow the customers and developers to simplify
>>>>>>> collection
>>>>>>> of log events statistics and searching/filtering of specific
events
>> for
>>>>>>> debugging and investigation.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sergey
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov <v.rozov@datatorrent.com
>>>>>>> wrote:
>>>>>>>
>>>>>>> +1 Apex engine does not own log4j config file - it is provided
either
>>>>>>> by
>>>>>>>
>>>>>>>> Hadoop or an application. Hadoop log4j config does not necessarily
>>>>>>>> meet
>>>>>>>> application logging requirements, but if log4j is provided
by an
>>>>>>>> application designer, who can only specify what to log, it
may not
>>>>>>>> meet
>>>>>>>> operations requirements. Dev-ops should have an ability to
specify
>>>>>>>> where
>>>>>>>>
>>>>>>>> to
>>>>>>> log depending on the available infrastructure at run-time.
>>>>>>>> It will be good to have an ability not only specify extra
log4j
>>>>>>>> appenders
>>>>>>>> at lunch time, but also at run-time, the same way how log4j
logger
>>>>>>>> levels
>>>>>>>> may be changed.
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> Vlad
>>>>>>>>
>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote:
>>>>>>>>
>>>>>>>> We can always write a custom appender and add it by changing
root
>>>>>>>> appender
>>>>>>>> in log4j config file. Can you explain how adding appender
>>>>>>>> grammatically
>>>>>>>>
>>>>>>>>> would help?
>>>>>>>>>
>>>>>>>>> -Priyanka
>>>>>>>>>
>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
>>>>>>>>> sanjay@datatorrent.com
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Please give some examples and/or use cases of this programmatic
>> log4j
>>>>>>>>> appender.
>>>>>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
>>>>>>>>>> sergey@datatorrent.com
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> I'd like to add supporting of a custom defined log4j
appender that
>>>>>>>>>>> can
>>>>>>>>>>> be
>>>>>>>>>>> added to Apex Application Master and Containers
and be
>> configurable
>>>>>>>>>>> programmatically.
>>>>>>>>>>>
>>>>>>>>>>> Sometimes it is not trivial to control log4j
configuration via
>>>>>>>>>>> log4j
>>>>>>>>>>> properties. And I think the having of the approach
to add a log4j
>>>>>>>>>>>
>>>>>>>>>>> appender
>>>>>>>>>>>
>>>>>>>>>> programmatically will allow the customers and developers
to plugin
>>>>>>>>>> their
>>>>>>>>>>
>>>>>>>>> own custom defined log4j appenders and be much flexible
for
>> streaming
>>>>>>>>> and
>>>>>>>>>>> collection of Apex log events.
>>>>>>>>>>>
>>>>>>>>>>> I assume to provide generic approach for definition
of the
>>>>>>>>>>>
>>>>>>>>>>> programmatic
>>>>>>>>> log4j appender and to pass all configuration parameters
including a
>>>>>>>>> name
>>>>>>>>> of
>>>>>>>>> the Java class with implementation of the log4j appender
via system
>>>>>>>>>>> and/or
>>>>>>>>>>>
>>>>>>>>>> command line properties.
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Sergey
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>
>


Mime
View raw message