Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
From: Jason Smith <jhs@iriscouch.com>
Date: Wed, 7 Dec 2011 08:51:37 +0700
Message-ID: 
 <CAN-3CBJzD2E0-acYy0itaia7A=RjR4A1FkCTzz=QgratJOewRg@mail.gmail.com>
Subject: The perfect logger for development
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=UTF-8

Hi, all. Iris Couch urgently needs improved logging facilities in
CouchDB. My goal is to make something we all love and get it accepted
upstream, God willing. Or committers willing. But I repeat myself!

This is the brainstorming and requirements gathering phase. In the
CouchDB of your dreams, logging system fits you like an old pair of
sneakers. It's perfect. Now, what characteristics does that system
exhibit? I will compile feedback into a spec on the wiki.

I hope to avoid bikeshedding. Seriously, please don't even mention a
product or project by name. At this phase I hope to stick to
descriptions of functionality, goals, and non-goals. I want to
evaluate tools later.

To start the discussion: logging is viewed differently based on your
relationship with CouchDB:

1. Developers of CouchDB
2. Administrators of a couch
3. Application developers

My roles are administration, and a little bit of development.
Requirements, in no order.

* Idiomatic Erlang

* Is a compelling place for new people to contribute. Miguel de Icaza
talks about this. It's not enough that the source code is public. You
have to provide a smooth on-ramp, where people people get great bang
for their buck. They write a modest feature, and are rewarded by
seeing its effects immediately. In other words: plugins. Or maybe a
behaviour. Or some way to swap in formatters and data sinks. I don't
want to write a Loggly target (http://loggly.com). Loggly should be
begging me to merge their module.

* 1st cut, no change to the default behavior. You still get the that
peculiar log file you know and love. People are parsing their log
files, and might expect version 1.x not to change.

* Existing code still works. No sweeping changes hitting every
?LOG_INFO and ?LOG_DEBUG.

(Filipe, would you please share your thoughts on these? I think you
struggled with the conflict between them recently.)
* No performance impact (non-blocking)...
* ... but also, impossible or difficult to overwhelm or crash or lose logs.

(The next few points sort of fit together)

* Logs are not strings, but data structures, with data (the log
message) and metadata (severity, line number, maybe the call stack,
etc.)

* More log levels. Roughly: trace, debug, info, warn, error, fatal

* Maybe automatic trace logs upon function entry/exit with return
values. Not sure if this is doable. Maybe a compile option, for `make
dev`

* When you log something, your module, line number, and maybe PID are known

* "Components" or categories, or tags, where multiple .erl files or
individual log events can share a common property ("http", "views",

* A policy decides what to do with logs based on level, module, or
component. You can set the policy either via configuration or
programatically.

* There is a formatter API to serialize log data. Built-in formatters
include the legacy format, and Jan's Apache httpd combined format.

* There is a transport API to receive logs and DTRT.

* I know this is insane, but kill -HUP <pid> should make couch reopen
its log files. Okay, I'll settle down now.

= Non Goals =

* Log rotation. I have never seen a rotation feature in an application
which was better than the OS tools. And you can get problem where both
the server and the OS are rotating the same logs. I have seen that
happen, twice. Or was it once? Of course, people could write a
rotating file transport.

-- 
Iris Couch