apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Programmatic log4j appender in Apex
Date Thu, 13 Apr 2017 14:56:23 GMT
+1

Also the proposed feature would need to be implemented in a way that avoids
a hard dependency on log4j. The interface for logging is slf4j and it
should be possible to use other logger backends.


On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko <sergey@datatorrent.com>
wrote:

> I don't think an operator needs a specific appender. An appender can be
> dynamically assigned to an application designer, application master and
> container.
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath <ram@datatorrent.com>
> wrote:
>
> > I don't have one, I thought that was what the intent of the proposal was,
> > but looks like
> > I misunderstood. After re-reading some of the earlier responses, I
> > understand the
> > proposal better.
> >
> > Ram
> >
> >
> >
> > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov <v.rozov@datatorrent.com>
> > wrote:
> >
> > > I don't see a use case where an individual operators need to define a
> > > specific appender, can you provide one?
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > > On 4/10/17 16:53, Munagala Ramanath wrote:
> > >
> > >> Yes, totally agree, it would be helpful to have a detailed use case
> > and/or
> > >> a detailed spec
> > >> of the desired capabilities -- not necessarily a complete spec but
> with
> > >> enough detail to
> > >> understand why existing capabilities are inadequate.
> > >>
> > >> Ram
> > >>
> > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <v.rozov@datatorrent.com>
> > >> wrote:
> > >>
> > >> It will be good to understand a use case where an operator needs a
> > >>> specific appender.
> > >>>
> > >>> IMO, an operator designer defines *what* should be logged and dev-ops
> > >>> team
> > >>> defines *where* to log.
> > >>>
> > >>> Thank you,
> > >>>
> > >>> Vlad
> > >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> > >>>
> > >>> Yes, I understand, I was just wondering if individual operators could
> > >>>> define the appenders
> > >>>> they potentially need at compile time and then the operator
> callbacks
> > >>>> could
> > >>>> simply
> > >>>> check the desired runtime condition and add the appropriate
> appender.
> > >>>>
> > >>>> Or are we saying there are scenarios where we absolutely cannot
> create
> > >>>> the
> > >>>> appender beforehand ?
> > >>>>
> > >>>> So broadly speaking, my question is whether the combination of
> > providing
> > >>>> predefined appenders
> > >>>> and the PropertyConfigurator capabilities meets the need.
> > >>>>
> > >>>> Ram
> > >>>>
> > >>>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> > sergey@datatorrent.com
> > >>>> >
> > >>>> wrote:
> > >>>>
> > >>>> Ram,
> > >>>>
> > >>>>> Really the new appender class must extend the abstract class
> > >>>>> AppenderSkeleton. And in order to add a new appender
> programmatically
> > >>>>> in
> > >>>>> Java, some code in Apex should call the following log4j method:
> > >>>>>
> > >>>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> > >>>>> newAppender)
> > >>>>>
> > >>>>> The general idea of my proposal is "*based on some runtime
> > parameter(s)
> > >>>>> to
> > >>>>> provide ability to create an appender instance via reflection
and
> add
> > >>>>> it
> > >>>>> to
> > >>>>> the list of active log4j appenders*".
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Sergey
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <
> v.rozov@datatorrent.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>> It will require application recompilation and repackaging.
The
> > proposed
> > >>>>>
> > >>>>>> functionality is for dev-ops to be able to route application
> logging
> > >>>>>> to
> > >>>>>> a
> > >>>>>> preferred destination without recompiling applications.
It is
> > run-time
> > >>>>>> configuration vs compile time hardcoded appender.
> > >>>>>>
> > >>>>>> Thank you,
> > >>>>>>
> > >>>>>> Vlad
> > >>>>>>
> > >>>>>> On 4/10/17 11:23, Munagala Ramanath wrote:
> > >>>>>>
> > >>>>>> You can do it in a trivial derived class without changing
the base
> > >>>>>> class.
> > >>>>>> Ram
> > >>>>>>
> > >>>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <
> > >>>>>>> v.rozov@datatorrent.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Does not the proposal to use Logger.addAppender() requires
> > >>>>>>> modifications
> > >>>>>>>
> > >>>>>>> to used operators code?
> > >>>>>>>>
> > >>>>>>>> Thank you,
> > >>>>>>>>
> > >>>>>>>> Vlad
> > >>>>>>>>
> > >>>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote:
> > >>>>>>>>
> > >>>>>>>> People can currently do this by simply implementing
the Appender
> > >>>>>>>>
> > >>>>>>>> interface
> > >>>>>>>>> and adding it
> > >>>>>>>>> with Logger.addAppender() in the setup method.
Why do we need
> > >>>>>>>>>
> > >>>>>>>>> something
> > >>>>>>>>
> > >>>>>>> more elaborate ?
> > >>>>>>
> > >>>>>>> Ram
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko
<
> > >>>>>>>>> sergey@datatorrent.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> The configuration of a log4j appender via log4j
configuration
> > file
> > >>>>>>>>> is
> > >>>>>>>>>
> > >>>>>>>>> a
> > >>>>>>>>
> > >>>>>>> static configuration that cannot be disabled/enabled
and managed
> > >>>>>>
> > >>>>>>> dynamically by an application designer. The programmatic
approach
> > >>>>>>>>>>
> > >>>>>>>>>> will
> > >>>>>>>>>
> > >>>>>>>> allow  an application designer to specify which
of the available
> > >>>>>>
> > >>>>>>> log4j
> > >>>>>>>>>
> > >>>>>>>> appenders should be used for the specific application.
> > >>>>>>
> > >>>>>>> It is not necessary Apex should use the predefined
log4j
> appenders
> > >>>>>>>>>> only.
> > >>>>>>>>>> The log4j events contain useful but the
very limited number of
> > >>>>>>>>>> properties
> > >>>>>>>>>> which values can be printed into output
log4j sources. But
> based
> > >>>>>>>>>> on
> > >>>>>>>>>>
> > >>>>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>> knowledge of the software product workflow, the
custom defined
> > log4j
> > >>>>>>
> > >>>>>>> appender can extend a list of predefined output log
events
> > >>>>>>>>>> properties
> > >>>>>>>>>> and,
> > >>>>>>>>>> for instance for Apex, return: node, user
name, application
> > name,
> > >>>>>>>>>> application id, container id, operator
name, etc.
> > >>>>>>>>>>
> > >>>>>>>>>> Also the output log events that are generated
by a custom
> > defined
> > >>>>>>>>>>
> > >>>>>>>>>> log4j
> > >>>>>>>>>
> > >>>>>>>> appender can be stored and indexed by any type
of a full text
> > search
> > >>>>>>
> > >>>>>>> database. It will allow the customers and developers
to simplify
> > >>>>>>>>>> collection
> > >>>>>>>>>> of log events statistics and searching/filtering
of specific
> > >>>>>>>>>> events
> > >>>>>>>>>>
> > >>>>>>>>>> for
> > >>>>>>>>>
> > >>>>>>>> debugging and investigation.
> > >>>>>>
> > >>>>>>> Thanks,
> > >>>>>>>>>> Sergey
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov
<
> > >>>>>>>>>> v.rozov@datatorrent.com
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> +1 Apex engine does not own log4j config
file - it is provided
> > >>>>>>>>>> either
> > >>>>>>>>>> by
> > >>>>>>>>>>
> > >>>>>>>>>> Hadoop or an application. Hadoop log4j
config does not
> > necessarily
> > >>>>>>>>>>
> > >>>>>>>>>>> meet
> > >>>>>>>>>>> application logging requirements, but
if log4j is provided by
> > an
> > >>>>>>>>>>> application designer, who can only
specify what to log, it
> may
> > >>>>>>>>>>> not
> > >>>>>>>>>>> meet
> > >>>>>>>>>>> operations requirements. Dev-ops should
have an ability to
> > >>>>>>>>>>> specify
> > >>>>>>>>>>> where
> > >>>>>>>>>>>
> > >>>>>>>>>>> to
> > >>>>>>>>>>>
> > >>>>>>>>>>> log depending on the available infrastructure
at run-time.
> > >>>>>>>>>>
> > >>>>>>>>>> It will be good to have an ability not
only specify extra
> log4j
> > >>>>>>>>>>> appenders
> > >>>>>>>>>>> at lunch time, but also at run-time,
the same way how log4j
> > >>>>>>>>>>> logger
> > >>>>>>>>>>> levels
> > >>>>>>>>>>> may be changed.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thank you,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Vlad
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> We can always write a custom appender
and add it by changing
> > root
> > >>>>>>>>>>> appender
> > >>>>>>>>>>> in log4j config file. Can you explain
how adding appender
> > >>>>>>>>>>> grammatically
> > >>>>>>>>>>>
> > >>>>>>>>>>> would help?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> -Priyanka
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM,
Sanjay Pujare <
> > >>>>>>>>>>>> sanjay@datatorrent.com
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Please give some examples and/or
use cases of this
> > programmatic
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> log4j
> > >>>>>>>>>>>
> > >>>>>>>>>> appender.
> > >>>>>>
> > >>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
> > >>>>>>>>>>>>> sergey@datatorrent.com
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi All,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'd like to add supporting
of a custom defined log4j
> appender
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> can
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>> added to Apex Application
Master and Containers and be
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> configurable
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> programmatically.
> > >>>>>>
> > >>>>>>> Sometimes it is not trivial to control log4j configuration
via
> > >>>>>>>>>>>>>> log4j
> > >>>>>>>>>>>>>> properties. And I think
the having of the approach to add
> a
> > >>>>>>>>>>>>>> log4j
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> appender
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> programmatically will allow
the customers and developers
> to
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> plugin
> > >>>>>>>>>>>>> their
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> own custom defined log4j appenders
and be much flexible for
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> streaming
> > >>>>>>>>>>>
> > >>>>>>>>>> and
> > >>>>>>
> > >>>>>>> collection of Apex log events.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I assume to provide generic
approach for definition of the
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> programmatic
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> log4j appender and to pass
all configuration parameters
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> including a
> > >>>>>>>>>>>> name
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>> the Java class with implementation
of the log4j appender via
> > >>>>>>>>>>>> system
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> and/or
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> command line properties.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Sergey
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>
> > >>
> > >
> >
> >
> > --
> >
> > _______________________________________________________
> >
> > Munagala V. Ramanath
> >
> > Software Engineer
> >
> > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> >
> > www.datatorrent.com  |  apex.apache.org
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message