lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gus Heck (JIRA)" <>
Subject [jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
Date Thu, 01 Feb 2018 06:57:00 GMT


Gus Heck commented on SOLR-11934:

First off, let me say I LIKE the heavy logging. When the $#!7 hits the fan, if someone tells
me when it went wrong, with a default Solr install I have something to dig into to find out
what went wrong. Often the answer is something like: "because you had two mutually exclusive
filters selected" or "the record was added but not committed yet" or "That's because the
system paused for a stop the world GC for 40 minutes" or "That looks like a pagination bug
in the UI, they added a filter but didn't reset the pagination" or "the term you're searching
for is in a field that's not indexed" Many of these things rely on knowing *exactly* what
the query or order of events was from *Solr's* perspective (particularly vs what the customer
+_thought_+ they sent for a query).

The case of a "very large system" where logging is burdensome is IMHO an edge case. Perhaps
some sort of ready made profiles for this edge case should exist but there are almost always
a whole lot more little fish than big fish in the sea.

That said, yes, the level of logging should be appropriate.  Since you asked for opinions,
here's my opinion on levels:
 * FATAL - worthy of immediate attention (beeper, SMS, whatever),  zoo keeper unreachable,
system fails to come up etc.
 * ERROR - Something serious, should be looked at in the morning or at least some time soon,
may be prelude to FATAL showing up, or high likelihood of customer bug report, very rarely
something that can be ignored.
 * WARN - Something questionable, possible backlog ticket, possibly ignore/turn off in some
 * INFO - that which is likely to be useful figure out what led up to the ERROR that woke
you at 3am... Also that which will be helpful in trouble shooting customer bug reports (yesterday
at we got this empty result when there were definitely documents that should have matched....turns
out to be a pagination bug). This definitely includes queries, updates, admin commands, gc
logs etc.
 * DEBUG - That which might help in trouble shooting or finding a bug, or trouble shooting
ane odd behavior. Rarely by admins, usually by devs
 * TRACE - Stuff only ever turned on by devs missing tufts of hair trying to figure out things
like "what order did those 2 threads run X and Y in"

Possibly a separate ticket but related: Breaking things out into separate files with potentially
different rotation frequencies, verbosities etc. seems like a decent idea. I only rarely want
to see admin/updates/queries together, and none of those alongside system logging... GC is
already separated, which is excellent. IMHO Turning off logging of queries/updates/whatever
up front is premature optimization and presents a usability issue. Splitting things into
more focused files probably helps most folks (except those who have been enjoying the firehose
with log analysis tools I suppose) but that's just my opinion. If things are broken up it
should be possible to restore the firehose for back compatibility.

The lack of query/update logs out of the box is one of the things that irritates me about
Elastic FWIW.

Obviously it would be a different ticket, but maybe some additional admin UI controls  to
augment the huge tree of loggers would be good? I'm thinking broad presets for default/quiet/verbose
for each type of log. 


> Visit Solr logging, it's too noisy.
> -----------------------------------
>                 Key: SOLR-11934
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Major
> I think we have way too much INFO level logging. Or, perhaps more correctly, Solr logging
needs to be examined and messages logged at an appropriate level.
> We log every update at an INFO level for instance. But I think we log LIR at INFO as
well. As a sysadmin I don't care to have my logs polluted with a message for every update,
but if I'm trying to keep my system healthy I want to see LIR messages and try to understand
> Plus, in large installations logging at INFO level is creating a _LOT_ of files.
> What I want to discuss on this JIRA is
> 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE levels?
> 2> Who's the audience at each level? For a running system that's functioning, sysops
folks would really like WARN messages that mean something need attention for instance. If
I'm troubleshooting should I turn on INFO? DEBUG? TRACE?
> So let's say we get some kind of agreement as to the above. Then I propose three things
> 1> Someone (and probably me but all help gratefully accepted) needs to go through
our logging and assign appropriate levels. This will take quite a while, I intend to work
on it in small chunks.
> 2> Actually answer whether unnecessary objects are created when something like"whatever
{}", someObjectOrMethodCall); is invoked. Is this independent on the logging implementation
used? The SLF4J and log4j seem a bit contradictory.
> 3> Maybe regularize log, logger, LOG as variable names, but that's a nit.
> As a tactical approach, I suggest we tag each LoggerFactory.getLogger in files we work
on with //SOLR-(whatever number is assigned when I create this). We can remove them all later,
but since I expect to approach this piecemeal it'd be nice to keep track of which files have
been done already.
> Finally, I really really really don't want to do this all at once. There are 5-6 thousand
log messages. Even at 1,000 a week that's 6 weeks, even starting now it would probably span
the 7.3 release.
> This will probably be an umbrella issue so we can keep all the commits straight and people
can volunteer to "fix the files in core" as a separate piece of work (hint).
> There are several existing JIRAs about logging in general, let's link them in here as
> Let the discussion begin!

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message