hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Reworking the use of log levels
Date Sat, 25 Oct 2014 19:42:44 GMT
That sounds great!

Either a shell or hbase command would be my suggestion, preferably with
some logic for mapping from the short logger name that ends up in the log
to the actual logger to adjust.

The ref guide would need a section on how to use the proposed log changing
tool along based on seeing a log message.

Would being able to make the logger with the failure change at runtime to
show the stack trace address your concerns Andrew and Qiang?

To be clear, I'm not saying we should *never* have stack traces above
DEBUG, just that they should be rare and reserved for when we can't point
an operator at something to do.

-- 
Sean
On Oct 25, 2014 1:50 PM, "Nick Dimiduk" <ndimiduk@gmail.com> wrote:

> We have the ability to alter log levels at runtime. This would allow an
> operator to temporarily increase log level for afflicted components, even
> in production. Doing this on a server-by-server basis should have minimal
> impact on overall cluster performance. Maybe this needs to be better
> documented? Maybe we need a script that makes this easier, or could be
> managed via a new shell command?
>
> On Saturday, October 25, 2014, Andrew Purtell <apurtell@apache.org> wrote:
>
> > ​
> > On Sat, Oct 25, 2014 at 6:34 AM, Sean Busbey <busbey@cloudera.com
> > <javascript:;>> wrote:
> >
> > > Even if debug is disabled in production, it could be enabled on a
> > > non-production system for reproducing the problem, no?
> > >
> >
> > ​In my experience, often enough, no.​
> >
> > I do hear the complaint that Hadoop ecosystem projects are quite operator
> > unfriendly because error messages most often come in the form of a
> > stacktrace. It's a totally valid point. I think we could certainly
> improve
> > the exception message printed ahead of the stacktrace in a large number
> of
> > cases.
> >
> >
> >
> > On Sat, Oct 25, 2014 at 6:34 AM, Sean Busbey <busbey@cloudera.com
> > <javascript:;>> wrote:
> >
> > > Even if debug is disabled in production, it could be enabled on a
> > > non-production system for reproducing the problem, no?
> > >
> > > --
> > > Sean
> > > On Oct 25, 2014 7:11 AM, "Qiang Tian" <tianq01@gmail.com
> <javascript:;>>
> > wrote:
> > >
> > > > perhaps case by case is better. stacktrace is one of most important
> > > problem
> > > > determination methods.  debug is mostly disabled in production, we
> may
> > > lose
> > > > important clues.
> > > >
> > > >
> > > > On Sat, Oct 25, 2014 at 1:14 PM, Sean Busbey <busbey@cloudera.com
> > <javascript:;>>
> > > wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > Right now we have many failure paths where we send stack traces to
> > log
> > > > > files at ERROR / WARN. In an effort to make things easier to
> operate,
> > > I'd
> > > > > like to propose we move towards:
> > > > >
> > > > > * INFO/WARN/ERROR : description of failure and if possible an
> action
> > an
> > > > > operator could take to fix/diagnose
> > > > > * DEBUG : information needed to handle failures that require
> > developer
> > > > > action, i.e. stack traces
> > > > >
> > > > > I figure this can go as one or more subtasks off of HBASE-12341,
> but
> > > > wanted
> > > > > to float things here before I get started.
> > > > >
> > > > > --
> > > > > Sean
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message