cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: kvm: /var/log/cloudstack/agent/agent.log is a binary file
Date Thu, 05 Sep 2019 09:13:21 GMT
Wido, makes sense that log4j and logrotate would conflict. Log4j has its
own rotate functionality.
I didn't understand before but the binary data is always the beginning of
the file? Is always nulls?

Op do 5 sep. 2019 09:18 schreef Wei ZHOU <ustcweizhou@gmail.com>:

> Hi Wido,
>
> I saw this issue in a 4.11.3 platform today.
>
> It seems to be caused by file
>
> https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in
>
> Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu
> ?).
>
> -Wei
>
>
>
> On Thu, 5 Sep 2019 at 08:08, Wido den Hollander <wido@widodh.nl> wrote:
>
> >
> >
> > On 9/3/19 2:55 PM, Daan Hoogland wrote:
> > > On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander <wido@widodh.nl>
> > wrote:
> > >
> > >>
> > >>
> > >> On 9/3/19 9:57 AM, Daan Hoogland wrote:
> > >>> Can you find/look at the line before in the log. It is probably the
> one
> > >>> containing the hindering data. Or otherwise it *might* be a clue
> where
> > in
> > >>> the flow it happens.
> > >>>
> > >>
> > >> Do you have any idea what the easiest way might be?
> > >>
> > > In short, no, but.. in totally skewed order of alleged quickness
> > > 1: I'm afraid it is a line by line analysis. If you are lucky the line
> > > start with date and class name is not obscured and there is a newline
> > > entered in the log. That would give you the exact location. when in bad
> > > luck the binary data contains control chars that erase to beginning of
> > line
> > > or so and the best you can find is the previous and next lines.
> > > 3: code analysis; My first thought is that some binary data is read
> from
> > a
> > > file or a socket and logged as is. but starting this with code analysis
> > is
> > > not the quickest way to get to the culprit I imagine.
> > > 2: Other than line by line log analysis and if you can reproduce the
> > issue,
> > > you could try zooming in by playing around with log levels for
> different
> > > classes.
> > >
> >
> > I checked, but on all hypervisors I checked the "agent.log" contains
> > binary data, but all the rotated (.gz) files do not.
> >
> > The binary data is at the beginning of the file(s). So it seems like it
> > is logrotate and maybe a combination of log4j causing this?
> >
> > I'm not sure yet, but that seems to be the case.
> >
> > Wido
> >
> > >
> > >>
> > >> I checked agent.log.1.gz and after I decompress is that file is not
> > >> binary, text only.
> > >>
> > > so much for reproducibilty. Hope it was not a buffer overflow hack
> > attempt.
> > >
> > >
> > >>
> > >> The binary data only seems to be present in the current file.
> > >>
> > >> Wido
> > >>
> > >>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander <wido@widodh.nl>
> > >> wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>> On 9/2/19 10:18 PM, Wei ZHOU wrote:
> > >>>>> Hi Wido,
> > >>>>>
> > >>>>> I had similar issue (not agent.log). It is probably caused
by one
> or
> > >>>> few> lines with special characters.
> > >>>>
> > >>>> And I'm trying to figure out what causes it :-)
> > >>>>
> > >>>>> "grep -a" should work.
> > >>>>
> > >>>> I know, but other tools which analyze the logfile might also have
> > >>>> trouble reading the file due to binary characters.
> > >>>>
> > >>>> Wido
> > >>>>
> > >>>>>
> > >>>>> -Wei
> > >>>>>
> > >>>>> On Mon, 2 Sep 2019 at 19:35, Wido den Hollander <wido@widodh.nl>
> > >> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> I've seen this on multiple occasions with Ubuntu 18.04
(and maybe
> > >>>>>> 16.04?) hypervisors where according to 'grep' the agent.log
is a
> > >> binary
> > >>>>>> file:
> > >>>>>>
> > >>>>>> root@n06:~# grep security_group
> /var/log/cloudstack/agent/agent.log
> > >>>>>> Binary file /var/log/cloudstack/agent/agent.log matches
> > >>>>>> root@n06:~#
> > >>>>>>
> > >>>>>> If I open the file with 'less' I indeed see binary data.
> > >>>>>>
> > >>>>>> Tailing the file works just fine.
> > >>>>>>
> > >>>>>> Does anybody know where this is coming from? What in the
> CloudStack
> > >>>>>> agent causes it to write binary data to the logfile?
> > >>>>>>
> > >>>>>> Wido
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message