Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
MIME-Version: 1.0
In-Reply-To: <CAFGBo=znmDCqKGfGgaPZmXskmNnHtYArfQLLS==MKuZr=oEoKQ@mail.gmail.com>
References: <CAFGBo=yWUBt1E+qj0OnHJ7QX98-nB+YknzD0GDaHVo_tsL=9DQ@mail.gmail.com>
 <CALPtS-JCunhUMdoMBDY-Abj9PzscndrmqXgRy=7ivK405u5OFw@mail.gmail.com>
 <CAKt=akEffgW63BO4eXmzQbvGZ+Jv37RNP2EFjJ67p_XJ3BcQ8A@mail.gmail.com>
 <0864a4d6-995f-e636-470c-60c1c43d0621@datatorrent.com> <CAFGBo=wZ9HBv3Y3qS3gqjuCt=j=xD95meO36MD8JShv42QZ1yg@mail.gmail.com>
 <CAJ484BgAZU3Zy2YG+DDw6Bw4rey9xuM8zLKtydAvmF52cr988Q@mail.gmail.com>
 <8e6122f3-8244-a7be-fbaa-0941b4697545@datatorrent.com> <CAJ484BgMXDoOa8ZeQ4BNDbGMMf9LUHZ-4AJWRd5envcymO3BKw@mail.gmail.com>
 <5afc31ff-0fb7-0473-f3f7-4b732f829b4b@datatorrent.com> <CAFGBo=zg3rD2ciy-ebMhCTf64pq-gkZ6R_AgMCJ67mTRTmgPxw@mail.gmail.com>
 <CAJ484Bg2bTwR=Q-L6Sd5DGqrJpNRdEVcW7fkUEs7cDpQ6E_DSw@mail.gmail.com>
 <f1deea19-85b1-0313-b7fd-3f0e6255fdca@datatorrent.com> <CAJ484Bic6Gp5hh=qMKOq9qSc1_XAbpQaM5uQ2cFf7oSv5MyX2w@mail.gmail.com>
 <f960efe6-2635-a578-c0bb-c8c4bd2f5392@datatorrent.com> <CAJ484BgtT-WNjA1ah++P1o6ORqiDevHsRL0dced3wbVieENcng@mail.gmail.com>
 <CAFGBo=znmDCqKGfGgaPZmXskmNnHtYArfQLLS==MKuZr=oEoKQ@mail.gmail.com>
From: Thomas Weise <thw@apache.org>
Date: Thu, 13 Apr 2017 07:56:23 -0700
Message-ID: <CA+5xAo08LjWoQp1w-EtJtctcS3=SYEATLqTLM+t1uFzjhPpwXQ@mail.gmail.com>
Subject: Re: Programmatic log4j appender in Apex
To: dev@apex.apache.org
Content-Type: multipart/alternative; boundary=94eb2c03c3b257a218054d0d8727
archived-at: Thu, 13 Apr 2017 14:56:27 -0000

--94eb2c03c3b257a218054d0d8727
Content-Type: text/plain; charset=UTF-8

+1

Also the proposed feature would need to be implemented in a way that avoids
a hard dependency on log4j. The interface for logging is slf4j and it
should be possible to use other logger backends.


On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko <sergey@datatorrent.com>
wrote:

> I don't think an operator needs a specific appender. An appender can be
> dynamically assigned to an application designer, application master and
> container.
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath <ram@datatorrent.com>
> wrote:
>
> > I don't have one, I thought that was what the intent of the proposal was,
> > but looks like
> > I misunderstood. After re-reading some of the earlier responses, I
> > understand the
> > proposal better.
> >
> > Ram
> >
> >
> >
> > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov <v.rozov@datatorrent.com>
> > wrote:
> >
> > > I don't see a use case where an individual operators need to define a
> > > specific appender, can you provide one?
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > > On 4/10/17 16:53, Munagala Ramanath wrote:
> > >
> > >> Yes, totally agree, it would be helpful to have a detailed use case
> > and/or
> > >> a detailed spec
> > >> of the desired capabilities -- not necessarily a complete spec but
> with
> > >> enough detail to
> > >> understand why existing capabilities are inadequate.
> > >>
> > >> Ram
> > >>
> > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <v.rozov@datatorrent.com>
> > >> wrote:
> > >>
> > >> It will be good to understand a use case where an operator needs a
> > >>> specific appender.
> > >>>
> > >>> IMO, an operator designer defines *what* should be logged and dev-ops
> > >>> team
> > >>> defines *where* to log.
> > >>>
> > >>> Thank you,
> > >>>
> > >>> Vlad
> > >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> > >>>
> > >>> Yes, I understand, I was just wondering if individual operators could
> > >>>> define the appenders
> > >>>> they potentially need at compile time and then the operator
> callbacks
> > >>>> could
> > >>>> simply
> > >>>> check the desired runtime condition and add the appropriate
> appender.
> > >>>>
> > >>>> Or are we saying there are scenarios where we absolutely cannot
> create
> > >>>> the
> > >>>> appender beforehand ?
> > >>>>
> > >>>> So broadly speaking, my question is whether the combination of
> > providing
> > >>>> predefined appenders
> > >>>> and the PropertyConfigurator capabilities meets the need.
> > >>>>
> > >>>> Ram
> > >>>>
> > >>>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> > sergey@datatorrent.com
> > >>>> >
> > >>>> wrote:
> > >>>>
> > >>>> Ram,
> > >>>>
> > >>>>> Really the new appender class must extend the abstract class
> > >>>>> AppenderSkeleton. And in order to add a new appender
> programmatically
> > >>>>> in
> > >>>>> Java, some code in Apex should call the following log4j method:
> > >>>>>
> > >>>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> > >>>>> newAppender)
> > >>>>>
> > >>>>> The general idea of my proposal is "*based on some runtime
> > parameter(s)
> > >>>>> to
> > >>>>> provide ability to create an appender instance via reflection and
> add
> > >>>>> it
> > >>>>> to
> > >>>>> the list of active log4j appenders*".
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Sergey
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <
> v.rozov@datatorrent.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>> It will require application recompilation and repackaging. The
> > proposed
> > >>>>>
> > >>>>>> functionality is for dev-ops to be able to route application
> logging
> > >>>>>> to
> > >>>>>> a
> > >>>>>> preferred destination without recompiling applications. It is
> > run-time
> > >>>>>> configuration vs compile time hardcoded appender.
> > >>>>>>
> > >>>>>> Thank you,
> > >>>>>>
> > >>>>>> Vlad
> > >>>>>>
> > >>>>>> On 4/10/17 11:23, Munagala Ramanath wrote:
> > >>>>>>
> > >>>>>> You can do it in a trivial derived class without changing the base
> > >>>>>> class.
> > >>>>>> Ram
> > >>>>>>
> > >>>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <
> > >>>>>>> v.rozov@datatorrent.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Does not the proposal to use Logger.addAppender() requires
> > >>>>>>> modifications
> > >>>>>>>
> > >>>>>>> to used operators code?
> > >>>>>>>>
> > >>>>>>>> Thank you,
> > >>>>>>>>
> > >>>>>>>> Vlad
> > >>>>>>>>
> > >>>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote:
> > >>>>>>>>
> > >>>>>>>> People can currently do this by simply implementing the Appender
> > >>>>>>>>
> > >>>>>>>> interface
> > >>>>>>>>> and adding it
> > >>>>>>>>> with Logger.addAppender() in the setup method. Why do we need
> > >>>>>>>>>
> > >>>>>>>>> something
> > >>>>>>>>
> > >>>>>>> more elaborate ?
> > >>>>>>
> > >>>>>>> Ram
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
> > >>>>>>>>> sergey@datatorrent.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> The configuration of a log4j appender via log4j configuration
> > file
> > >>>>>>>>> is
> > >>>>>>>>>
> > >>>>>>>>> a
> > >>>>>>>>
> > >>>>>>> static configuration that cannot be disabled/enabled and managed
> > >>>>>>
> > >>>>>>> dynamically by an application designer. The programmatic approach
> > >>>>>>>>>>
> > >>>>>>>>>> will
> > >>>>>>>>>
> > >>>>>>>> allow  an application designer to specify which of the available
> > >>>>>>
> > >>>>>>> log4j
> > >>>>>>>>>
> > >>>>>>>> appenders should be used for the specific application.
> > >>>>>>
> > >>>>>>> It is not necessary Apex should use the predefined log4j
> appenders
> > >>>>>>>>>> only.
> > >>>>>>>>>> The log4j events contain useful but the very limited number of
> > >>>>>>>>>> properties
> > >>>>>>>>>> which values can be printed into output log4j sources. But
> based
> > >>>>>>>>>> on
> > >>>>>>>>>>
> > >>>>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>> knowledge of the software product workflow, the custom defined
> > log4j
> > >>>>>>
> > >>>>>>> appender can extend a list of predefined output log events
> > >>>>>>>>>> properties
> > >>>>>>>>>> and,
> > >>>>>>>>>> for instance for Apex, return: node, user name, application
> > name,
> > >>>>>>>>>> application id, container id, operator name, etc.
> > >>>>>>>>>>
> > >>>>>>>>>> Also the output log events that are generated by a custom
> > defined
> > >>>>>>>>>>
> > >>>>>>>>>> log4j
> > >>>>>>>>>
> > >>>>>>>> appender can be stored and indexed by any type of a full text
> > search
> > >>>>>>
> > >>>>>>> database. It will allow the customers and developers to simplify
> > >>>>>>>>>> collection
> > >>>>>>>>>> of log events statistics and searching/filtering of specific
> > >>>>>>>>>> events
> > >>>>>>>>>>
> > >>>>>>>>>> for
> > >>>>>>>>>
> > >>>>>>>> debugging and investigation.
> > >>>>>>
> > >>>>>>> Thanks,
> > >>>>>>>>>> Sergey
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov <
> > >>>>>>>>>> v.rozov@datatorrent.com
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> +1 Apex engine does not own log4j config file - it is provided
> > >>>>>>>>>> either
> > >>>>>>>>>> by
> > >>>>>>>>>>
> > >>>>>>>>>> Hadoop or an application. Hadoop log4j config does not
> > necessarily
> > >>>>>>>>>>
> > >>>>>>>>>>> meet
> > >>>>>>>>>>> application logging requirements, but if log4j is provided by
> > an
> > >>>>>>>>>>> application designer, who can only specify what to log, it
> may
> > >>>>>>>>>>> not
> > >>>>>>>>>>> meet
> > >>>>>>>>>>> operations requirements. Dev-ops should have an ability to
> > >>>>>>>>>>> specify
> > >>>>>>>>>>> where
> > >>>>>>>>>>>
> > >>>>>>>>>>> to
> > >>>>>>>>>>>
> > >>>>>>>>>>> log depending on the available infrastructure at run-time.
> > >>>>>>>>>>
> > >>>>>>>>>> It will be good to have an ability not only specify extra
> log4j
> > >>>>>>>>>>> appenders
> > >>>>>>>>>>> at lunch time, but also at run-time, the same way how log4j
> > >>>>>>>>>>> logger
> > >>>>>>>>>>> levels
> > >>>>>>>>>>> may be changed.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thank you,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Vlad
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> We can always write a custom appender and add it by changing
> > root
> > >>>>>>>>>>> appender
> > >>>>>>>>>>> in log4j config file. Can you explain how adding appender
> > >>>>>>>>>>> grammatically
> > >>>>>>>>>>>
> > >>>>>>>>>>> would help?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> -Priyanka
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
> > >>>>>>>>>>>> sanjay@datatorrent.com
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Please give some examples and/or use cases of this
> > programmatic
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> log4j
> > >>>>>>>>>>>
> > >>>>>>>>>> appender.
> > >>>>>>
> > >>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
> > >>>>>>>>>>>>> sergey@datatorrent.com
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi All,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'd like to add supporting of a custom defined log4j
> appender
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> can
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>> added to Apex Application Master and Containers and be
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> configurable
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> programmatically.
> > >>>>>>
> > >>>>>>> Sometimes it is not trivial to control log4j configuration via
> > >>>>>>>>>>>>>> log4j
> > >>>>>>>>>>>>>> properties. And I think the having of the approach to add
> a
> > >>>>>>>>>>>>>> log4j
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> appender
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> programmatically will allow the customers and developers
> to
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> plugin
> > >>>>>>>>>>>>> their
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> own custom defined log4j appenders and be much flexible for
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> streaming
> > >>>>>>>>>>>
> > >>>>>>>>>> and
> > >>>>>>
> > >>>>>>> collection of Apex log events.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I assume to provide generic approach for definition of the
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> programmatic
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> log4j appender and to pass all configuration parameters
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> including a
> > >>>>>>>>>>>> name
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>> the Java class with implementation of the log4j appender via
> > >>>>>>>>>>>> system
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> and/or
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> command line properties.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Sergey
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>
> > >>
> > >
> >
> >
> > --
> >
> > _______________________________________________________
> >
> > Munagala V. Ramanath
> >
> > Software Engineer
> >
> > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> >
> > www.datatorrent.com  |  apex.apache.org
> >
>

--94eb2c03c3b257a218054d0d8727--