apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Programmatic log4j appender in Apex
Date Mon, 10 Apr 2017 23:53:40 GMT
Yes, totally agree, it would be helpful to have a detailed use case and/or
a detailed spec
of the desired capabilities -- not necessarily a complete spec but with
enough detail to
understand why existing capabilities are inadequate.

Ram

On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> It will be good to understand a use case where an operator needs a
> specific appender.
>
> IMO, an operator designer defines *what* should be logged and dev-ops team
> defines *where* to log.
>
> Thank you,
>
> Vlad
> On 4/10/17 16:27, Munagala Ramanath wrote:
>
>> Yes, I understand, I was just wondering if individual operators could
>> define the appenders
>> they potentially need at compile time and then the operator callbacks
>> could
>> simply
>> check the desired runtime condition and add the appropriate appender.
>>
>> Or are we saying there are scenarios where we absolutely cannot create the
>> appender beforehand ?
>>
>> So broadly speaking, my question is whether the combination of providing
>> predefined appenders
>> and the PropertyConfigurator capabilities meets the need.
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <sergey@datatorrent.com>
>> wrote:
>>
>> Ram,
>>>
>>> Really the new appender class must extend the abstract class
>>> AppenderSkeleton. And in order to add a new appender programmatically in
>>> Java, some code in Apex should call the following log4j method:
>>>
>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender
>>> newAppender)
>>>
>>> The general idea of my proposal is "*based on some runtime parameter(s)
>>> to
>>> provide ability to create an appender instance via reflection and add it
>>> to
>>> the list of active log4j appenders*".
>>>
>>> Thanks,
>>> Sergey
>>>
>>>
>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <v.rozov@datatorrent.com>
>>> wrote:
>>>
>>> It will require application recompilation and repackaging. The proposed
>>>> functionality is for dev-ops to be able to route application logging to
>>>> a
>>>> preferred destination without recompiling applications. It is run-time
>>>> configuration vs compile time hardcoded appender.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>> On 4/10/17 11:23, Munagala Ramanath wrote:
>>>>
>>>> You can do it in a trivial derived class without changing the base
>>>>>
>>>> class.
>>>
>>>> Ram
>>>>>
>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <v.rozov@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> Does not the proposal to use Logger.addAppender() requires
>>>>> modifications
>>>>>
>>>>>> to used operators code?
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Vlad
>>>>>>
>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote:
>>>>>>
>>>>>> People can currently do this by simply implementing the Appender
>>>>>>
>>>>>>> interface
>>>>>>> and adding it
>>>>>>> with Logger.addAppender() in the setup method. Why do we need
>>>>>>>
>>>>>> something
>>>
>>>> more elaborate ?
>>>>>>>
>>>>>>> Ram
>>>>>>>
>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
>>>>>>> sergey@datatorrent.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> The configuration of a log4j appender via log4j configuration
file is
>>>>>>>
>>>>>> a
>>>
>>>> static configuration that cannot be disabled/enabled and managed
>>>>>>>> dynamically by an application designer. The programmatic
approach
>>>>>>>>
>>>>>>> will
>>>
>>>> allow  an application designer to specify which of the available
>>>>>>>>
>>>>>>> log4j
>>>
>>>> appenders should be used for the specific application.
>>>>>>>>
>>>>>>>> It is not necessary Apex should use the predefined log4j
appenders
>>>>>>>> only.
>>>>>>>> The log4j events contain useful but the very limited number
of
>>>>>>>> properties
>>>>>>>> which values can be printed into output log4j sources. But
based on
>>>>>>>>
>>>>>>> the
>>>
>>>> knowledge of the software product workflow, the custom defined log4j
>>>>>>>> appender can extend a list of predefined output log events
>>>>>>>> properties
>>>>>>>> and,
>>>>>>>> for instance for Apex, return: node, user name, application
name,
>>>>>>>> application id, container id, operator name, etc.
>>>>>>>>
>>>>>>>> Also the output log events that are generated by a custom
defined
>>>>>>>>
>>>>>>> log4j
>>>
>>>> appender can be stored and indexed by any type of a full text search
>>>>>>>> database. It will allow the customers and developers to simplify
>>>>>>>> collection
>>>>>>>> of log events statistics and searching/filtering of specific
events
>>>>>>>>
>>>>>>> for
>>>
>>>> debugging and investigation.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sergey
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov <
>>>>>>>> v.rozov@datatorrent.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> +1 Apex engine does not own log4j config file - it is provided
>>>>>>>> either
>>>>>>>> by
>>>>>>>>
>>>>>>>> Hadoop or an application. Hadoop log4j config does not necessarily
>>>>>>>>> meet
>>>>>>>>> application logging requirements, but if log4j is provided
by an
>>>>>>>>> application designer, who can only specify what to log,
it may not
>>>>>>>>> meet
>>>>>>>>> operations requirements. Dev-ops should have an ability
to specify
>>>>>>>>> where
>>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>> log depending on the available infrastructure at run-time.
>>>>>>>>
>>>>>>>>> It will be good to have an ability not only specify extra
log4j
>>>>>>>>> appenders
>>>>>>>>> at lunch time, but also at run-time, the same way how
log4j logger
>>>>>>>>> levels
>>>>>>>>> may be changed.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>>
>>>>>>>>> Vlad
>>>>>>>>>
>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote:
>>>>>>>>>
>>>>>>>>> We can always write a custom appender and add it by changing
root
>>>>>>>>> appender
>>>>>>>>> in log4j config file. Can you explain how adding appender
>>>>>>>>> grammatically
>>>>>>>>>
>>>>>>>>> would help?
>>>>>>>>>>
>>>>>>>>>> -Priyanka
>>>>>>>>>>
>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
>>>>>>>>>> sanjay@datatorrent.com
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Please give some examples and/or use cases of this
programmatic
>>>>>>>>>>
>>>>>>>>> log4j
>>>
>>>> appender.
>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko
<
>>>>>>>>>>> sergey@datatorrent.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I'd like to add supporting of a custom defined
log4j appender
>>>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> can
>>>>>>>>>>>> be
>>>>>>>>>>>> added to Apex Application Master and Containers
and be
>>>>>>>>>>>>
>>>>>>>>>>> configurable
>>>
>>>> programmatically.
>>>>>>>>>>>>
>>>>>>>>>>>> Sometimes it is not trivial to control log4j
configuration via
>>>>>>>>>>>> log4j
>>>>>>>>>>>> properties. And I think the having of the
approach to add a
>>>>>>>>>>>> log4j
>>>>>>>>>>>>
>>>>>>>>>>>> appender
>>>>>>>>>>>>
>>>>>>>>>>>> programmatically will allow the customers
and developers to
>>>>>>>>>>> plugin
>>>>>>>>>>> their
>>>>>>>>>>>
>>>>>>>>>>> own custom defined log4j appenders and be much
flexible for
>>>>>>>>>>
>>>>>>>>> streaming
>>>
>>>> and
>>>>>>>>>>
>>>>>>>>>>> collection of Apex log events.
>>>>>>>>>>>>
>>>>>>>>>>>> I assume to provide generic approach for
definition of the
>>>>>>>>>>>>
>>>>>>>>>>>> programmatic
>>>>>>>>>>>>
>>>>>>>>>>> log4j appender and to pass all configuration
parameters
>>>>>>>>>> including a
>>>>>>>>>> name
>>>>>>>>>> of
>>>>>>>>>> the Java class with implementation of the log4j appender
via
>>>>>>>>>> system
>>>>>>>>>>
>>>>>>>>>>> and/or
>>>>>>>>>>>>
>>>>>>>>>>>> command line properties.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Sergey
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>
>>
>


-- 

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message