apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AJAY GUPTA <ajaygit...@gmail.com>
Subject Re: Adding new log4j appender to Apex core
Date Wed, 23 Nov 2016 07:05:05 GMT
+1 for approach 2.


Regards,
Ajay

On Wed, Nov 23, 2016 at 12:16 PM, David Yan <david@datatorrent.com> wrote:

> The goal of this log4j appender is to provide a log offset and the fixed
> name of the container log file (instead of apex.log becoming apex.log.1 and
> then apex.log.2, etc due to rotation) as part of an error STRAM event so
> users can easily locate the log entries around the error.
>
> The user can override the appender, but in that case, the engine detects
> that and will not include the log location as part of the STRAM event.
>
> David
>
> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <priyanka@datatorrent.com
> >
> wrote:
>
> > Hi,
> >
> > Thomas,
> > Yes log4j is ultimately owned by user, and they should be able to
> override
> > it. What I am trying to do is provide a default behavior for Apex. In
> case
> > user isn't using any logger of their own we should use this new appender
> of
> > Apex rather than using standard log4j appender as per hadoop config.
> >
> > Sanjay,
> > Archetype is the good place to put this and I will add it there, but many
> > time people won't use it. So I wanted to keep it at ~/.dt as well. Is
> there
> > any other default config folder for Apex?
> >
> > Also I am not relying on anything. If we fail to find config in app jar
> or
> > ~/.dt we are going to skip usage of this new appender.
> >
> > -Priyanka
> >
> > On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <sanjay@datatorrent.com>
> > wrote:
> >
> > > The only way to “enforce” this new appender is to update the archetypes
> > > (apex-app-archetype and apex-conf-archetype under apex-core/ )  to use
> > the
> > > new ones as default. But there does not seem to be a way to enforce
> this
> > > for anyone not using the archetypes.
> > >
> > > I agree with not relying on ~/.dt in apex-core.
> > >
> > > On 11/22/16, 1:08 PM, "Thomas Weise" <thw@apache.org> wrote:
> > >
> > >     The log4j configuration is ultimately owned by the user, so how do
> > you
> > > want
> > >     to enforce a custom appender?
> > >
> > >     I don't think that this should rely on anything in ~/.dt either
> > >
> > >     Thomas
> > >
> > >     On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> > > priyanka@datatorrent.com>
> > >     wrote:
> > >
> > >     > Hi,
> > >     >
> > >     > I am working on APEXCORE-563
> > >     > <https://issues.apache.org/jira/browse/APEXCORE-563>
> > >     > As per this Jira we should put log file name in
> container/operator
> > > events.
> > >     > The problem is current RollingFileAppender keeps renaming files
> > from
> > > 1 to 2
> > >     > to ... n as files reach maximum allowed file size. Because of
> > > constant
> > >     > renaming of files we can't put a fixed file name in stram event.
> > >     >
> > >     > To overcome this I would like to add a new log4j appender to
> > > ApexCore.
> > >     > There are two ways I can implement this:
> > >     > 1. Have Daily rolling file appender. The current file will be
> > > recognized
> > >     > based on timestamp in file name. Also to control max file size,
> we
> > > need to
> > >     > keep rolling files based on size as well.
> > >     > 2. Have Rolling File Appender but do not rename files. When max
> > file
> > > size
> > >     > is reached create new file with next number e.g. crate log file
> > > dt.log.2
> > >     > after dt.log.1 is full. Also to recognize the latest file keep
> the
> > > softlink
> > >     > named dt.log pointing to current log file.
> > >     >
> > >     > I would prefer to implement approach 2. Please provide your
> > >     > comments/feedback if you feel otherwise.
> > >     >
> > >     > Also to use this new appender we need to use our custom
> > > log4j.properties
> > >     > file instead of one present in hadoop conf. For that we need to
> set
> > > jvm
> > >     > option -Dlog4j.configuration. I am planning to update file
> > > dt-site.xml in
> > >     > folder ~/.dt  and default properties file available in apex
> > > archetype to
> > >     > set jvm options as follows:
> > >     >  <property>
> > >     >    <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> > >     >    <value>-Dlog4j.configuration=log4j.props</value>
> > >     >  </property>
> > >     >
> > >     > And I will copy log4j.props file in ~/.dt folder as well as
> > >     > apex-archetypes.
> > >     >
> > >     > Lastly if someone still miss this new log4j properties file or
> jvm
> > > option
> > >     > to set -Dlog4j.configuration we will not put log file name in
> event
> > > raised
> > >     > by container or operator.
> > >     >
> > >     > Please provide your feedback on this approach.
> > >     >
> > >     > -Priyanka
> > >     >
> > >
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message