Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
MIME-Version: 1.0
In-Reply-To: <5afc31ff-0fb7-0473-f3f7-4b732f829b4b@datatorrent.com>
References: <CAFGBo=yWUBt1E+qj0OnHJ7QX98-nB+YknzD0GDaHVo_tsL=9DQ@mail.gmail.com>
 <CALPtS-JCunhUMdoMBDY-Abj9PzscndrmqXgRy=7ivK405u5OFw@mail.gmail.com>
 <CAKt=akEffgW63BO4eXmzQbvGZ+Jv37RNP2EFjJ67p_XJ3BcQ8A@mail.gmail.com>
 <0864a4d6-995f-e636-470c-60c1c43d0621@datatorrent.com> <CAFGBo=wZ9HBv3Y3qS3gqjuCt=j=xD95meO36MD8JShv42QZ1yg@mail.gmail.com>
 <CAJ484BgAZU3Zy2YG+DDw6Bw4rey9xuM8zLKtydAvmF52cr988Q@mail.gmail.com>
 <8e6122f3-8244-a7be-fbaa-0941b4697545@datatorrent.com> <CAJ484BgMXDoOa8ZeQ4BNDbGMMf9LUHZ-4AJWRd5envcymO3BKw@mail.gmail.com>
 <5afc31ff-0fb7-0473-f3f7-4b732f829b4b@datatorrent.com>
From: Munagala Ramanath <ram@datatorrent.com>
Date: Mon, 10 Apr 2017 16:18:09 -0700
Message-ID: <CAJ484Bis7fhCjYdEeBy7sEYPNxfcoL1MRoKGduNt6T7H3AOW9w@mail.gmail.com>
Subject: Re: Programmatic log4j appender in Apex
To: dev@apex.apache.org
Content-Type: multipart/alternative; boundary=001a113f84da415fea054cd830fe
archived-at: Mon, 10 Apr 2017 23:18:17 -0000

--001a113f84da415fea054cd830fe
Content-Type: text/plain; charset=UTF-8

http://logging.apache.org/log4j/1.2/faq.html#3.6

Log4j has the ability to dynamically reload changes to the properties file
via the
*configureAndWatch()* method of *PropertyConfigurator*. If this is already
baked in, could
devops simply change the properties file if we provide an enhanced set of
appenders
for the desired use cases ?

Ram

On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> It will require application recompilation and repackaging. The proposed
> functionality is for dev-ops to be able to route application logging to a
> preferred destination without recompiling applications. It is run-time
> configuration vs compile time hardcoded appender.
>
> Thank you,
>
> Vlad
> On 4/10/17 11:23, Munagala Ramanath wrote:
>
>> You can do it in a trivial derived class without changing the base class.
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <v.rozov@datatorrent.com>
>> wrote:
>>
>> Does not the proposal to use Logger.addAppender() requires modifications
>>> to used operators code?
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 4/10/17 10:58, Munagala Ramanath wrote:
>>>
>>> People can currently do this by simply implementing the Appender
>>>> interface
>>>> and adding it
>>>> with Logger.addAppender() in the setup method. Why do we need something
>>>> more elaborate ?
>>>>
>>>> Ram
>>>>
>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
>>>> sergey@datatorrent.com>
>>>> wrote:
>>>>
>>>> The configuration of a log4j appender via log4j configuration file is a
>>>>
>>>>> static configuration that cannot be disabled/enabled and managed
>>>>> dynamically by an application designer. The programmatic approach will
>>>>> allow  an application designer to specify which of the available log4j
>>>>> appenders should be used for the specific application.
>>>>>
>>>>> It is not necessary Apex should use the predefined log4j appenders
>>>>> only.
>>>>> The log4j events contain useful but the very limited number of
>>>>> properties
>>>>> which values can be printed into output log4j sources. But based on the
>>>>> knowledge of the software product workflow, the custom defined log4j
>>>>> appender can extend a list of predefined output log events properties
>>>>> and,
>>>>> for instance for Apex, return: node, user name, application name,
>>>>> application id, container id, operator name, etc.
>>>>>
>>>>> Also the output log events that are generated by a custom defined log4j
>>>>> appender can be stored and indexed by any type of a full text search
>>>>> database. It will allow the customers and developers to simplify
>>>>> collection
>>>>> of log events statistics and searching/filtering of specific events for
>>>>> debugging and investigation.
>>>>>
>>>>> Thanks,
>>>>> Sergey
>>>>>
>>>>>
>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov <v.rozov@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> +1 Apex engine does not own log4j config file - it is provided either
>>>>> by
>>>>>
>>>>>> Hadoop or an application. Hadoop log4j config does not necessarily
>>>>>> meet
>>>>>> application logging requirements, but if log4j is provided by an
>>>>>> application designer, who can only specify what to log, it may not
>>>>>> meet
>>>>>> operations requirements. Dev-ops should have an ability to specify
>>>>>> where
>>>>>>
>>>>>> to
>>>>>
>>>>> log depending on the available infrastructure at run-time.
>>>>>>
>>>>>> It will be good to have an ability not only specify extra log4j
>>>>>> appenders
>>>>>> at lunch time, but also at run-time, the same way how log4j logger
>>>>>> levels
>>>>>> may be changed.
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Vlad
>>>>>>
>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote:
>>>>>>
>>>>>> We can always write a custom appender and add it by changing root
>>>>>> appender
>>>>>> in log4j config file. Can you explain how adding appender
>>>>>> grammatically
>>>>>>
>>>>>>> would help?
>>>>>>>
>>>>>>> -Priyanka
>>>>>>>
>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
>>>>>>> sanjay@datatorrent.com
>>>>>>> wrote:
>>>>>>>
>>>>>>> Please give some examples and/or use cases of this programmatic log4j
>>>>>>>
>>>>>>> appender.
>>>>>>>>
>>>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
>>>>>>>> sergey@datatorrent.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I'd like to add supporting of a custom defined log4j appender that
>>>>>>>>> can
>>>>>>>>> be
>>>>>>>>> added to Apex Application Master and Containers and be configurable
>>>>>>>>> programmatically.
>>>>>>>>>
>>>>>>>>> Sometimes it is not trivial to control log4j configuration via
>>>>>>>>> log4j
>>>>>>>>> properties. And I think the having of the approach to add a log4j
>>>>>>>>>
>>>>>>>>> appender
>>>>>>>>>
>>>>>>>> programmatically will allow the customers and developers to plugin
>>>>>>>> their
>>>>>>>>
>>>>>>> own custom defined log4j appenders and be much flexible for streaming
>>>>>>
>>>>>>> and
>>>>>>>>> collection of Apex log events.
>>>>>>>>>
>>>>>>>>> I assume to provide generic approach for definition of the
>>>>>>>>>
>>>>>>>>> programmatic
>>>>>>>>
>>>>>>> log4j appender and to pass all configuration parameters including a
>>>>>>
>>>>>>> name
>>>>>>>>
>>>>>>> of
>>>>>>
>>>>>>> the Java class with implementation of the log4j appender via system
>>>>>>>>
>>>>>>>>> and/or
>>>>>>>>>
>>>>>>>> command line properties.
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Sergey
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>
>>
>


-- 

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org

--001a113f84da415fea054cd830fe--