Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
MIME-Version: 1.0
In-Reply-To: <f1deea19-85b1-0313-b7fd-3f0e6255fdca@datatorrent.com>
References: <CAFGBo=yWUBt1E+qj0OnHJ7QX98-nB+YknzD0GDaHVo_tsL=9DQ@mail.gmail.com>
 <CALPtS-JCunhUMdoMBDY-Abj9PzscndrmqXgRy=7ivK405u5OFw@mail.gmail.com>
 <CAKt=akEffgW63BO4eXmzQbvGZ+Jv37RNP2EFjJ67p_XJ3BcQ8A@mail.gmail.com>
 <0864a4d6-995f-e636-470c-60c1c43d0621@datatorrent.com> <CAFGBo=wZ9HBv3Y3qS3gqjuCt=j=xD95meO36MD8JShv42QZ1yg@mail.gmail.com>
 <CAJ484BgAZU3Zy2YG+DDw6Bw4rey9xuM8zLKtydAvmF52cr988Q@mail.gmail.com>
 <8e6122f3-8244-a7be-fbaa-0941b4697545@datatorrent.com> <CAJ484BgMXDoOa8ZeQ4BNDbGMMf9LUHZ-4AJWRd5envcymO3BKw@mail.gmail.com>
 <5afc31ff-0fb7-0473-f3f7-4b732f829b4b@datatorrent.com> <CAFGBo=zg3rD2ciy-ebMhCTf64pq-gkZ6R_AgMCJ67mTRTmgPxw@mail.gmail.com>
 <CAJ484Bg2bTwR=Q-L6Sd5DGqrJpNRdEVcW7fkUEs7cDpQ6E_DSw@mail.gmail.com> <f1deea19-85b1-0313-b7fd-3f0e6255fdca@datatorrent.com>
From: Munagala Ramanath <ram@datatorrent.com>
Date: Mon, 10 Apr 2017 16:53:40 -0700
Message-ID: <CAJ484Bic6Gp5hh=qMKOq9qSc1_XAbpQaM5uQ2cFf7oSv5MyX2w@mail.gmail.com>
Subject: Re: Programmatic log4j appender in Apex
To: dev@apex.apache.org
Content-Type: multipart/alternative; boundary=001a1144a9624e07b6054cd8af91
archived-at: Mon, 10 Apr 2017 23:53:48 -0000

--001a1144a9624e07b6054cd8af91
Content-Type: text/plain; charset=UTF-8

Yes, totally agree, it would be helpful to have a detailed use case and/or
a detailed spec
of the desired capabilities -- not necessarily a complete spec but with
enough detail to
understand why existing capabilities are inadequate.

Ram

On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> It will be good to understand a use case where an operator needs a
> specific appender.
>
> IMO, an operator designer defines *what* should be logged and dev-ops team
> defines *where* to log.
>
> Thank you,
>
> Vlad
> On 4/10/17 16:27, Munagala Ramanath wrote:
>
>> Yes, I understand, I was just wondering if individual operators could
>> define the appenders
>> they potentially need at compile time and then the operator callbacks
>> could
>> simply
>> check the desired runtime condition and add the appropriate appender.
>>
>> Or are we saying there are scenarios where we absolutely cannot create the
>> appender beforehand ?
>>
>> So broadly speaking, my question is whether the combination of providing
>> predefined appenders
>> and the PropertyConfigurator capabilities meets the need.
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <sergey@datatorrent.com>
>> wrote:
>>
>> Ram,
>>>
>>> Really the new appender class must extend the abstract class
>>> AppenderSkeleton. And in order to add a new appender programmatically in
>>> Java, some code in Apex should call the following log4j method:
>>>
>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender
>>> newAppender)
>>>
>>> The general idea of my proposal is "*based on some runtime parameter(s)
>>> to
>>> provide ability to create an appender instance via reflection and add it
>>> to
>>> the list of active log4j appenders*".
>>>
>>> Thanks,
>>> Sergey
>>>
>>>
>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <v.rozov@datatorrent.com>
>>> wrote:
>>>
>>> It will require application recompilation and repackaging. The proposed
>>>> functionality is for dev-ops to be able to route application logging to
>>>> a
>>>> preferred destination without recompiling applications. It is run-time
>>>> configuration vs compile time hardcoded appender.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>> On 4/10/17 11:23, Munagala Ramanath wrote:
>>>>
>>>> You can do it in a trivial derived class without changing the base
>>>>>
>>>> class.
>>>
>>>> Ram
>>>>>
>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <v.rozov@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> Does not the proposal to use Logger.addAppender() requires
>>>>> modifications
>>>>>
>>>>>> to used operators code?
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Vlad
>>>>>>
>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote:
>>>>>>
>>>>>> People can currently do this by simply implementing the Appender
>>>>>>
>>>>>>> interface
>>>>>>> and adding it
>>>>>>> with Logger.addAppender() in the setup method. Why do we need
>>>>>>>
>>>>>> something
>>>
>>>> more elaborate ?
>>>>>>>
>>>>>>> Ram
>>>>>>>
>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
>>>>>>> sergey@datatorrent.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> The configuration of a log4j appender via log4j configuration file is
>>>>>>>
>>>>>> a
>>>
>>>> static configuration that cannot be disabled/enabled and managed
>>>>>>>> dynamically by an application designer. The programmatic approach
>>>>>>>>
>>>>>>> will
>>>
>>>> allow  an application designer to specify which of the available
>>>>>>>>
>>>>>>> log4j
>>>
>>>> appenders should be used for the specific application.
>>>>>>>>
>>>>>>>> It is not necessary Apex should use the predefined log4j appenders
>>>>>>>> only.
>>>>>>>> The log4j events contain useful but the very limited number of
>>>>>>>> properties
>>>>>>>> which values can be printed into output log4j sources. But based on
>>>>>>>>
>>>>>>> the
>>>
>>>> knowledge of the software product workflow, the custom defined log4j
>>>>>>>> appender can extend a list of predefined output log events
>>>>>>>> properties
>>>>>>>> and,
>>>>>>>> for instance for Apex, return: node, user name, application name,
>>>>>>>> application id, container id, operator name, etc.
>>>>>>>>
>>>>>>>> Also the output log events that are generated by a custom defined
>>>>>>>>
>>>>>>> log4j
>>>
>>>> appender can be stored and indexed by any type of a full text search
>>>>>>>> database. It will allow the customers and developers to simplify
>>>>>>>> collection
>>>>>>>> of log events statistics and searching/filtering of specific events
>>>>>>>>
>>>>>>> for
>>>
>>>> debugging and investigation.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sergey
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov <
>>>>>>>> v.rozov@datatorrent.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> +1 Apex engine does not own log4j config file - it is provided
>>>>>>>> either
>>>>>>>> by
>>>>>>>>
>>>>>>>> Hadoop or an application. Hadoop log4j config does not necessarily
>>>>>>>>> meet
>>>>>>>>> application logging requirements, but if log4j is provided by an
>>>>>>>>> application designer, who can only specify what to log, it may not
>>>>>>>>> meet
>>>>>>>>> operations requirements. Dev-ops should have an ability to specify
>>>>>>>>> where
>>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>> log depending on the available infrastructure at run-time.
>>>>>>>>
>>>>>>>>> It will be good to have an ability not only specify extra log4j
>>>>>>>>> appenders
>>>>>>>>> at lunch time, but also at run-time, the same way how log4j logger
>>>>>>>>> levels
>>>>>>>>> may be changed.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>>
>>>>>>>>> Vlad
>>>>>>>>>
>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote:
>>>>>>>>>
>>>>>>>>> We can always write a custom appender and add it by changing root
>>>>>>>>> appender
>>>>>>>>> in log4j config file. Can you explain how adding appender
>>>>>>>>> grammatically
>>>>>>>>>
>>>>>>>>> would help?
>>>>>>>>>>
>>>>>>>>>> -Priyanka
>>>>>>>>>>
>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
>>>>>>>>>> sanjay@datatorrent.com
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Please give some examples and/or use cases of this programmatic
>>>>>>>>>>
>>>>>>>>> log4j
>>>
>>>> appender.
>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
>>>>>>>>>>> sergey@datatorrent.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I'd like to add supporting of a custom defined log4j appender
>>>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> can
>>>>>>>>>>>> be
>>>>>>>>>>>> added to Apex Application Master and Containers and be
>>>>>>>>>>>>
>>>>>>>>>>> configurable
>>>
>>>> programmatically.
>>>>>>>>>>>>
>>>>>>>>>>>> Sometimes it is not trivial to control log4j configuration via
>>>>>>>>>>>> log4j
>>>>>>>>>>>> properties. And I think the having of the approach to add a
>>>>>>>>>>>> log4j
>>>>>>>>>>>>
>>>>>>>>>>>> appender
>>>>>>>>>>>>
>>>>>>>>>>>> programmatically will allow the customers and developers to
>>>>>>>>>>> plugin
>>>>>>>>>>> their
>>>>>>>>>>>
>>>>>>>>>>> own custom defined log4j appenders and be much flexible for
>>>>>>>>>>
>>>>>>>>> streaming
>>>
>>>> and
>>>>>>>>>>
>>>>>>>>>>> collection of Apex log events.
>>>>>>>>>>>>
>>>>>>>>>>>> I assume to provide generic approach for definition of the
>>>>>>>>>>>>
>>>>>>>>>>>> programmatic
>>>>>>>>>>>>
>>>>>>>>>>> log4j appender and to pass all configuration parameters
>>>>>>>>>> including a
>>>>>>>>>> name
>>>>>>>>>> of
>>>>>>>>>> the Java class with implementation of the log4j appender via
>>>>>>>>>> system
>>>>>>>>>>
>>>>>>>>>>> and/or
>>>>>>>>>>>>
>>>>>>>>>>>> command line properties.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Sergey
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>
>>
>


-- 

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org

--001a1144a9624e07b6054cd8af91--