nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject Re: Logback stops writing to nifi-app.log after restart
Date Thu, 27 Oct 2016 19:59:22 GMT
Ahh, that would explain it!  Maybe we can change the comment in logback.xml
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml>
above this element so it's more clear:

<!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory>

On Thu, Oct 27, 2016 at 2:14 PM, Joe Witt <joe.witt@gmail.com> wrote:

> Joe,
>
> I believe that maxHistory is not 'how many files' to keep but 'how
> long to keep them for' where the length of time you specify is a
> function of the rollover interval.  So, if you have hourly rollover as
> you showed previously and if you have a value of '10' it will be
> trying to keep around up to 10 hours worth of rolled over files which
> could be a lot more than 10.  I talked to Dr Google and these [1][2]
> are why I'm saying that plus I've been confused by this behavior in
> the past.
>
> [1] http://stackoverflow.com/questions/7934829/logback-set-
> max-history-files-per-day
> [2] http://logback.qos.ch/manual/appenders.html#tbrpMaxHistory
>
> Thanks
> Joe
>
> On Thu, Oct 27, 2016 at 9:35 AM, Joe Gresock <jgresock@gmail.com> wrote:
> > It was not during the file handle exhaustion issu.  However, t I imagine
> it
> > has to do with logback not being happy with that filename format, since I
> > just restarted with %i and it now writes to the log upon restart.
> >
> > To demonstrate the rollover behavior, I changed <maxFileSize> to 10MB and
> > started generating millions of flow files.  Note that <maxHistory> is
> still
> > set to 10:
> >
> > -rw-rw-r--. 1 nifi nifi 104866674 Oct 27 13:09
> > nifi-app_2016-10-27_13.0.log      # This one was left over from when I
> had
> > 100MB configured
> > -rw-r--r--. 1 nifi nifi  10496507 Oct 27 13:31
> nifi-app_2016-10-27_13.10.log
> > -rw-r--r--. 1 nifi nifi  10494043 Oct 27 13:31
> nifi-app_2016-10-27_13.11.log
> > -rw-r--r--. 1 nifi nifi  10499265 Oct 27 13:31
> nifi-app_2016-10-27_13.12.log
> > -rw-r--r--. 1 nifi nifi  10500566 Oct 27 13:31
> nifi-app_2016-10-27_13.13.log
> > -rw-r--r--. 1 nifi nifi  10501235 Oct 27 13:31
> nifi-app_2016-10-27_13.14.log
> > -rw-r--r--. 1 nifi nifi  10496136 Oct 27 13:31
> nifi-app_2016-10-27_13.15.log
> > -rw-r--r--. 1 nifi nifi  10495439 Oct 27 13:31
> nifi-app_2016-10-27_13.16.log
> > -rw-r--r--. 1 nifi nifi  10500211 Oct 27 13:31
> nifi-app_2016-10-27_13.17.log
> > -rw-r--r--. 1 nifi nifi  10501075 Oct 27 13:31
> nifi-app_2016-10-27_13.18.log
> > -rw-r--r--. 1 nifi nifi  10497417 Oct 27 13:31
> nifi-app_2016-10-27_13.19.log
> > -rw-rw-r--. 1 nifi nifi  45261434 Oct 27 13:29
> > nifi-app_2016-10-27_13.1.log    # So is this one
> > -rw-r--r--. 1 nifi nifi  10496496 Oct 27 13:31
> nifi-app_2016-10-27_13.2.log
> > -rw-r--r--. 1 nifi nifi  10494287 Oct 27 13:31
> nifi-app_2016-10-27_13.3.log
> > -rw-r--r--. 1 nifi nifi  10495554 Oct 27 13:31
> nifi-app_2016-10-27_13.4.log
> > -rw-r--r--. 1 nifi nifi  10495347 Oct 27 13:31
> nifi-app_2016-10-27_13.5.log
> > -rw-r--r--. 1 nifi nifi  10501632 Oct 27 13:31
> nifi-app_2016-10-27_13.6.log
> > -rw-r--r--. 1 nifi nifi  10501246 Oct 27 13:31
> nifi-app_2016-10-27_13.7.log
> > -rw-r--r--. 1 nifi nifi  10500133 Oct 27 13:31
> nifi-app_2016-10-27_13.8.log
> > -rw-r--r--. 1 nifi nifi  10500095 Oct 27 13:31
> nifi-app_2016-10-27_13.9.log
> > -rw-rw-r--. 1 nifi nifi 104872440 Oct 27 13:04 nifi-app_2016-10-27_13.log
> > -rw-r--r--. 1 nifi nifi   9416149 Oct 27 13:31 nifi-app.log
> >
> > It appears that logback is not respecting the max file size and generates
> > 20 historical log files here.
> >
> >
> > On Thu, Oct 27, 2016 at 1:11 PM, Joe Witt <joe.witt@gmail.com> wrote:
> >
> >> if this was during the file handling exhaustion issue you'll see a lot
> >> of undesirable behaviors.  If this is happening when file handles are
> >> not a problem then it will require deeper digging.  That said, the
> >> base configuration does indeed write to nifi-app.log and roll over in
> >> the expected manner (at least in all cases I've seen).
> >>
> >> On Thu, Oct 27, 2016 at 9:07 AM, Joe Gresock <jgresock@gmail.com>
> wrote:
> >> > I'll try that, though I believe we removed the .%i because it
> continued
> >> to
> >> > make logs more than the configured <maxHistory>10</maxHistory>
when
> >> rolling
> >> > over, which ran us out of disk space.
> >> >
> >> > On Thu, Oct 27, 2016 at 1:00 PM, Joe Witt <joe.witt@gmail.com> wrote:
> >> >
> >> >> Joe
> >> >>
> >> >> I suspect you want to get closer to the original entry with the %i
> >> >> part.  So perhaps
> >> >>
> >> >> <fileNamePattern>/opt/nifi/data/logs/nifi-app_%d{yyyy-MM-
> >> dd_HH}.%i.log</
> >> >> fileNamePattern>
> >> >>
> >> >> Thanks
> >> >> Joe
> >> >>
> >> >> On Thu, Oct 27, 2016 at 8:57 AM, Joe Gresock <jgresock@gmail.com>
> >> wrote:
> >> >> > 23c23
> >> >> > <
> >> >> > <file>${org.apache.nifi.bootstrap.config.log.dir}/
> nifi-app.log</file>
> >> >> > ---
> >> >> >>         <file>/opt/nifi/data/logs/nifi-app.log</file>
> >> >> > 31c31
> >> >> > <
> >> >> > <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}
> >> >> /nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
> >> >> > ---
> >> >> >>
> >> >> > <fileNamePattern>/opt/nifi/data/logs/nifi-app_%d{yyyy-MM-
> >> >> dd_HH}.log</fileNamePattern>
> >> >> > 36c36,55
> >> >> > <             <maxHistory>30</maxHistory>
> >> >> > ---
> >> >> >>             <maxHistory>10</maxHistory>
> >> >> >
> >> >> >
> >> >> > On Thu, Oct 27, 2016 at 12:39 PM, Joe Witt <joe.witt@gmail.com>
> >> wrote:
> >> >> >
> >> >> >> Joe
> >> >> >>
> >> >> >> Can you highlight any changes (diff) from the standard
> configuration
> >> >> >> provided?
> >> >> >>
> >> >> >> Thanks
> >> >> >> Joe
> >> >> >>
> >> >> >> On Thu, Oct 27, 2016 at 8:34 AM, Joe Gresock <jgresock@gmail.com>
> >> >> wrote:
> >> >> >> > Before I create a ticket, I wanted to see if anyone had
run into
> >> this.
> >> >> >> >
> >> >> >> > Whenever we restart nifi, the nifi-app.log no longer
gets
> written
> >> to.
> >> >> >> >
> >> >> >> > If I delete all the nifi-app* logs and then restart,
it writes
> logs
> >> >> just
> >> >> >> > fine, but if I leave the logs it doesn't write to the
log upon
> >> >> restarting
> >> >> >> > nifi.
> >> >> >> >
> >> >> >> > Sure enough, "lsof | grep nifi-app" produces output prior
to
> >> restart,
> >> >> but
> >> >> >> > produces nothing after a restart (if I have not deleted
the
> logs).
> >> >> >> >
> >> >> >> > An excerpt from my logback.xml:
> >> >> >> >
> >> >> >> > <configuration scan="true" scanPeriod="30 seconds">
> >> >> >> >     <contextListener
> >> >> >> > class="ch.qos.logback.classic.jul.LevelChangePropagator">
> >> >> >> >         <resetJUL>true</resetJUL>
> >> >> >> >     </contextListener>
> >> >> >> >
> >> >> >> >     <appender name="APP_FILE"
> >> >> >> > class="ch.qos.logback.core.rolling.RollingFileAppender">
> >> >> >> >         <file>/opt/nifi/data/logs/nifi-app.log</file>
> >> >> >> >         <rollingPolicy
> >> >> >> > class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> >> >> >> >             <!--
> >> >> >> >               For daily rollover, use 'app_%d.log'.
> >> >> >> >               For hourly rollover, use
> 'app_%d{yyyy-MM-dd_HH}.log'.
> >> >> >> >               To GZIP rolled files, replace '.log' with
> '.log.gz'.
> >> >> >> >               To ZIP rolled files, replace '.log' with
> '.log.zip'.
> >> >> >> >             -->
> >> >> >> >
> >> >> >> > <fileNamePattern>/opt/nifi/data/logs/nifi-app_%d{yyyy-MM-
> >> >> >> dd_HH}.log</fileNamePattern>
> >> >> >> >             <timeBasedFileNamingAndTriggeringPolicy
> >> >> >> > class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
> >> >> >> >                 <maxFileSize>100MB</maxFileSize>
> >> >> >> >             </timeBasedFileNamingAndTriggeringPolicy>
> >> >> >> >             <maxHistory>10</maxHistory>
> >> >> >> >         </rollingPolicy>
> >> >> >> >         <encoder
> >> >> >> > class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
> >> >> >> >             <pattern>%date %level [%thread] %logger{40}
> >> >> %msg%n</pattern>
> >> >> >> >             <immediateFlush>true</immediateFlush>
> >> >> >> >         </encoder>
> >> >> >> >     </appender>
> >> >> >> >     <root level="INFO">
> >> >> >> >         <appender-ref ref="APP_FILE"/>
> >> >> >> >     </root>
> >> >> >> > </configuration>
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > I know what it is to be in need, and I know what it is
to have
> >> >> plenty.  I
> >> >> >> > have learned the secret of being content in any and every
> >> situation,
> >> >> >> > whether well fed or hungry, whether living in plenty
or in
> want.  I
> >> >> can
> >> >> >> do
> >> >> >> > all this through him who gives me strength.    *-Philippians
> >> 4:12-13*
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > I know what it is to be in need, and I know what it is to have
> >> plenty.  I
> >> >> > have learned the secret of being content in any and every
> situation,
> >> >> > whether well fed or hungry, whether living in plenty or in want.
 I
> >> can
> >> >> do
> >> >> > all this through him who gives me strength.    *-Philippians
> 4:12-13*
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > I know what it is to be in need, and I know what it is to have
> plenty.  I
> >> > have learned the secret of being content in any and every situation,
> >> > whether well fed or hungry, whether living in plenty or in want.  I
> can
> >> do
> >> > all this through him who gives me strength.    *-Philippians 4:12-13*
> >>
> >
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message